You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/08/12 08:20:48 UTC
[GitHub] [flink-connector-dynamodb] nirtsruya opened a new pull request, #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
nirtsruya opened a new pull request, #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1
Move flink-connector-aws-connector from the flink respository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998664686
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/config/AWSDynamoDbConfigConstants.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.config;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+
+import software.amazon.awssdk.core.retry.backoff.EqualJitterBackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.FixedDelayBackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.FullJitterBackoffStrategy;
+
+/** Defaults for {@link AWSDynamoDbUtil}. */
+@PublicEvolving
+public class AWSDynamoDbConfigConstants {
+
+ public static final String BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT =
+ "Apache Flink %s (%s) DynamoDB Connector";
+
+ /** Identifier for user agent prefix. */
+ public static final String DYNAMODB_CLIENT_USER_AGENT_PREFIX =
+ "aws.dynamodb.client.user-agent-prefix";
+
+ public static final String AWS_DYNAMODB_CLIENT_RETRY_PREFIX = "aws.dynamodb.client.retry";
+
+ /**
+ * Possible configuration values for backoff strategy on retry when writing to DynamoDB.
+ * Internally, a corresponding implementation of {@link
+ * software.amazon.awssdk.core.retry.backoff.BackoffStrategy} will be used.
+ */
+ @PublicEvolving
+ public enum BackoffStrategy {
+
+ /**
+ * Backoff strategy that uses a full jitter strategy for computing the next backoff delay. A
+ * full jitter strategy will always compute a new random delay between and the computed
+ * exponential backoff for each subsequent request. See example: {@link
+ * FullJitterBackoffStrategy}
+ */
+ FULL_JITTER,
+
+ /**
+ * Backoff strategy that uses equal jitter for computing the delay before the next retry. An
+ * equal jitter backoff strategy will first compute an exponential delay based on the
+ * current number of retries, base delay and max delay. The final computed delay before the
+ * next retry will keep half of this computed delay plus a random delay computed as a random
+ * number between 0 and half of the exponential delay plus one. See example: {@link
+ * EqualJitterBackoffStrategy}
+ */
+ EQUAL_JITTER,
+
+ /**
+ * Simple backoff strategy that always uses a fixed delay for the delay * before the next
+ * retry attempt. See example: {@link FixedDelayBackoffStrategy}
+ */
+ FIXED_DELAY
+ }
Review Comment:
Created a JIRA for it https://issues.apache.org/jira/browse/FLINK-29683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1012407031
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination table using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb. DynamoDB client supports only up to 25 elements in the batch.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * sink will backpressure while the number of elements in the buffer is at the maximum
+ * <li>{@code maxBatchSizeInBytes}: the maximum size of a batch of entries that may be written to
+ * DynamoDb measured in bytes
+ * <li>{@code maxTimeInBufferMS}: the maximum amount of time an entry is allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be flushed immediately
+ * <li>{@code maxRecordSizeInBytes}: the maximum size of a record the sink will accept into the
+ * buffer, a record of size larger than this will be rejected when passed to the sink
+ * <li>{@code failOnError}: when an exception is encountered while persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * <li>{@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in the same batch request.
+ * Only the latest record with the same combination of key attributes is preserved in the
+ * request.
+ * </ul>
Review Comment:
Fixed and other javadocs also updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-aws] hlteoh37 commented on pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on PR #1:
URL: https://github.com/apache/flink-connector-aws/pull/1#issuecomment-1304196612
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997909454
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Hi, I might be misunderstood the question.
Duplicated keys only matter when they get to the same batch request, then dynamodb service will fail the request. I took a simple approach to have accumulator to deduplicate entries based on PK/SK in the same request. E.g. if we had 25 entries and 1 entry has duplicated PK/SK we would then write 24 entries in the request.
But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000808803
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
+ // here, as we have at most 25 elements per batch
+ return UUID.randomUUID().toString();
+ } else {
+ Map<String, AttributeValue> requestItems = getRequestItems(request);
+
+ StringBuilder builder = new StringBuilder();
+ for (String keyName : pKeys) {
+ AttributeValue value = requestItems.get(keyName);
+
+ if (value == null) {
+ throw new InvalidRequestException(
+ "Request " + request.toString() + " does not contain pKey " + keyName);
+ }
+
+ builder.append(getKeyValue(value));
Review Comment:
very good point. Fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998328370
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
> I am also thinking, that if we want to really support that we probably have to remove that deduplication that we are doing in case we find a duplicate key, and instead "split" the batch, to make sure we do not lose records with the same primary key
IMO deduplication within a batch on the Flink DDB sink side is a useful feature, because it reduces writes to DDB. The current deduplication ensures the newest record takes precedence, which is the behaviour we see on consecutive `BatchWriteItem` requests.
The difference I can see here is that the ChangeDataCapture stream off the DDB table will be different depending on whether deduping happens in Flink DDB Sink. This might be possible if the user wants to write to DDB but also track number of changes in the stream. This sounds like a relatively niche use case though, since this use case is more suited to a Streaming Destination like Kinesis.
So, I think it makes sense to keep the deduping. @nirtsruya do you have a particular use case where splitting into batch would be preferable?
> But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
> If the order of the duplicate/update entries matters I think batch sink can not be used anyway, as the order is mixed up by partial write failures/full retries and parallelism in the sink. Then it is up to user to dedup before the sink.
As far as I can tell, the second request in DDB will take precedence and overwrite the first request. I can see Batching being very useful for Batch jobs where deduping already occured, and we just want to dump a large amt of data to DDB.
For streaming use case, this situation is mitigated if user sets the `MAX_IN_FLIGHT_REQUESTS` to 1, because the `AsyncSinkWriter` will put any failed items from requests to the front of the queue, so the "older" items will be retried first.
In the future, I was thinking we can implement an improvement to use `PutItem` API, and the DDB Sink can add an additional key-value with `eventTime` as the value. Whenever an item with older event time attempts to be written, we can just drop the older record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r966129384
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
We had a discussion earlier on that matter [here](https://github.com/apache/flink/pull/18518#discussion_r793711778)
If we go for hiding the converter we do not leak underlying dynamodb sdk classes to the user, but we have to create a wrapper around their model. This means if the dynamodb client changes over time we will have to propagate model changes too.
I think it may be better to allow the user to define the element converter and not do any translation ourselves (like we had it before this PR: https://github.com/YuriGusev/flink/commit/de83346dc24367ff1101ae13934dfcf1fc649de2#diff-1c18ee5b3c5b4d827778ffdf4e5d2219dbcb02fc45be0d3665357633a134995c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1008763133
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
Review Comment:
I decided to inline this class into the writer, as it was not doing much anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r996320784
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream using the buffering
Review Comment:
```suggestion
* A DynamoDB Sink that performs async requests against a destination table using the buffering
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997400104
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream using the buffering
Review Comment:
fixed thanks for noticing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003539179
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
Review Comment:
sorry missed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1014006791
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.aws.util.AWSAsyncSinkUtil;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.connector.base.sink.throwable.FatalExceptionClassifier;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getInvalidCredentialsExceptionClassifier;
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getSdkClientMisconfiguredExceptionClassifier;
+import static org.apache.flink.connector.base.sink.writer.AsyncSinkFatalExceptionClassifiers.getInterruptedExceptionClassifier;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ private static final FatalExceptionClassifier RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ResourceNotFoundException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to not being able to find the specified resources",
+ err));
+
+ private static final FatalExceptionClassifier CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ConditionalCheckFailedException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to failed conditional check",
+ err));
+
+ /* Validation exceptions are not retryable. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html. */
+ private static final FatalExceptionClassifier VALIDATION_EXCEPTION_CLASSIFIER =
+ new FatalExceptionClassifier(
+ (err) ->
+ err instanceof DynamoDbException
+ && ((DynamoDbException) err)
+ .toBuilder()
+ .awsErrorDetails()
+ .errorCode()
+ .equalsIgnoreCase("ValidationException"),
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception because of DynamoDB request validation",
+ err));
+
+ private static final FatalExceptionClassifier DYNAMODB_FATAL_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.createChain(
+ getInterruptedExceptionClassifier(),
+ getInvalidCredentialsExceptionClassifier(),
+ RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER,
+ CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER,
+ VALIDATION_EXCEPTION_CLASSIFIER,
+ getSdkClientMisconfiguredExceptionClassifier());
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* A counter for the total number of records that were returned by DynamoDB as unprocessed and were retried */
+ private final Counter numRecordsReturnedAsUnprocessedCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ /* The asynchronous http client for the asynchronous DynamoDb client */
+ private final SdkAsyncHttpClient httpClient;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private final List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPartitionKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.numRecordsReturnedAsUnprocessedCounter =
+ metrics.counter("numRecordsReturnedAsUnprocessedCounter");
+ this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
Review Comment:
```suggestion
dynamoDbClientProperties.put(AWSConfigConstants.HTTP_PROTOCOL_VERSION, "HTTP1_1");
this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#issuecomment-1279823905
Question for general discussion:
Do we have any thoughts on using conditional expressions? I note that the `BatchWriteItem` API we are using doesn't support this. Maybe we can create a followup JIRA to track this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998512587
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
May be we could provide a setting to turn off deduplication on the sink side. That would be on-pair with boto3 clients for dynamodb (overwrite_by_pkeys=['partition_key', 'sort_key'] config). Wdyt?
> In the future, I was thinking we can implement an improvement to use PutItem API, and the DDB Sink can add an additional key-value with eventTime as the value. Whenever an item with older event time attempts to be written, we can just drop the older record.
Yes I agree that would be a nice feature to add. We actually had it planned for [the first sink version](https://github.com/apache/flink/pull/17407/files#diff-ffcfaf3e245b27ca44831c37b908014be944626c1eac3d5acc63fa310345d937) (not based on AsyncSyncBase). There was interface that allows different types of producers (batching/non-batching) to be able to extend for this use case in the future. We can introduce something similar here and plan for adding this feature. For non-batching sink we could also support conditional expressions for individual PutItem/DeleteItem requests (as you mentioned below)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998651365
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
> May be we could provide a setting to turn off deduplication on the sink side
Currently we can do this via `tableConfig`, isn't it? Or do you think a separate config will be better? But I agree - providing a setting to toggle deduplication should be good enough
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
> May be we could provide a setting to turn off deduplication on the sink side
Currently we can do this via `tableConfig`, isn't it? Or do you think a separate config will be better? But I agree - providing a setting to toggle deduplication is a good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000808138
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbElementConverter implements ElementConverter<String, DynamoDbWriteRequest> {
+ *
+ * @Override
+ * public DynamoDbWriteRequest apply(String s) {
+ * final Map<String, AttributeValue> item = new HashMap<>();
+ * item.put("your-key", AttributeValue.builder().s(s).build());
+ * return new DynamoDbWriteRequest(
+ * WriteRequest.builder()
+ * .putRequest(PutRequest.builder()
+ * .item(item)
+ * .build())
+ * .build()
+ * );
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setElementConverter(new DummyDynamoDbElementConverter())
+ * .setDestinationTableName("your-table-name")
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code destinationTableName} destination table for the sink
+ * <li>{@code overwriteByPKeys} will be empty meaning no records deduplication will be performed
+ * by the batch sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
+ private static final int DEFAULT_MAX_BUFFERED_REQUESTS = 10000;
+ private static final long DEFAULT_MAX_BATCH_SIZE_IN_B = 16 * 1000 * 1000;
+ private static final long DEFAULT_MAX_TIME_IN_BUFFER_MS = 5000;
+ private static final long DEFAULT_MAX_RECORD_SIZE_IN_B = 400 * 1000;
+ private static final boolean DEFAULT_FAIL_ON_ERROR = false;
+
+ private boolean failOnError;
+ private Properties dynamodbClientProperties;
+
+ private ElementConverter<InputT, DynamoDbWriteRequest> elementConverter;
+ private String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkBuilder<InputT> setDynamoDbProperties(Properties properties) {
+ this.dynamodbClientProperties = properties;
+ return this;
+ }
+
+ public DynamoDbSinkBuilder<InputT> setElementConverter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter) {
+ this.elementConverter = elementConverter;
+ return this;
+ }
+
+ /** Destination DynamoDB table name for the sink to write to. */
+ public DynamoDbSinkBuilder<InputT> setDestinationTableName(String tableName) {
+ this.tableName = tableName;
+ return this;
+ }
+
+ /**
+ * @param overwriteByPKeys provide partition key and (optionally) sort key name if you want to
+ * bypass no duplication limitation of single batch write request. Batching DynamoDB sink
+ * will drop request items in the buffer if their primary keys(composite) values are the
+ * same as newly added one.
Review Comment:
We had it as a separate DTO, I changed it [here](https://github.com/apache/flink-connector-dynamodb/pull/1/commits/b51a075733618e6b9110e5253e9ee5450b4b6e8c#diff-d72f905550d45db880aa1a62971000bd9aeedf89a8f80a881da0e9266377d022). Order/number of keys doesn't matter for the deduplication logic. I can fix the comment to make it more clear, or return previous approach (or to have two individual fields as well).
I changed this is to match similar approach in the [boto3 dynamodb batch client](https://github.com/boto/boto3/blob/develop/boto3/dynamodb/table.py#L52).
Otherwise I'm fine to have two fields, however we would need to keep nullable field in the builder for sort key and pass both fields down the tree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1001914891
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequestType.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Represents a DynamoDb Write Request type. The following types are currently supported
+ *
+ * <ul>
+ * <li>PUT - Put Request
+ * <li>DELETE - Delete Request
+ * </ul>
+ */
+@PublicEvolving
+public enum DynamoDbWriteRequestType {
+ PUT,
+ DELETE,
+}
Review Comment:
Is it worth trying use a more general Flink type to represent this, something like https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/types/RowKind.java?
Maybe @MartijnVisser would be able to help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r994483063
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/config/DynamoDbTablesConfig.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.config;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/** Represents DynamoDB tables configuration. */
+@PublicEvolving
+public class DynamoDbTablesConfig implements Serializable {
Review Comment:
yes I agree this is over complication. I added support for it because the dynamodb client supports it. Agree it will be better to remove multiple tables support. I'll work on removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] sthm commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
sthm commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r995536272
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
Agree that having a wrapper provides limited benefits in this case and that it makes more sense to expose the DynamoDB SDK classes directly.
To mitigate the risk of not updating the de/serializers when doing a SDK update, it would be great if we could find a way to programmatically verify during compile time that the SDK is still compatible with the de/serializers. But I'm not sure how much work that would required and if it's worth the effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998632189
##########
flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/TableRequestsContainerTest.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/** Tests for {@link TableRequestsContainer}. */
+public class TableRequestsContainerTest {
+ @Test
+ public void testRequestNotDeduplicatedWhenNoTableConfig() {
+ new Scenario()
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .withDescription("number of requests without deduplication configuration")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testDeduplicatedOnCompositeKey() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk", "sk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ // to be deduplicated
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
Review Comment:
Good point I'll add test for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1002387162
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequestType.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Represents a DynamoDb Write Request type. The following types are currently supported
+ *
+ * <ul>
+ * <li>PUT - Put Request
+ * <li>DELETE - Delete Request
+ * </ul>
+ */
+@PublicEvolving
+public enum DynamoDbWriteRequestType {
+ PUT,
+ DELETE,
+}
Review Comment:
@dannycranmer I like using the byte representation for de/serialization. will also apply it for [DynamoDbType](https://github.com/apache/flink-connector-dynamodb/pull/1/files#diff-932831fb29c0c177a7b39476c623f21bea631a165797759d6f1c75c24d7ff49e)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000457255
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/config/DynamoDbTablesConfig.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.config;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/** Represents DynamoDB tables configuration. */
+@PublicEvolving
+public class DynamoDbTablesConfig implements Serializable {
Review Comment:
removed multiple tables support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r999023608
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
@hlteoh37 / @YuriGusev / @nirtsruya there has been discussed regarding support for Conditional Writes and non-batch support. I note that the batch API uses `PutRequest` and non-batch uses `PutItemRequest`. They both internally use the same `Map<String, AttributeValue>`, but the `PutItemRequest` adds additional fields.
My concern is that we would not be able to toggle batch mode without implementing a new connector, since the generic RequestItem type will change. Therefore, maybe we need some middle ground here. We could use a POJO that contains an `Map<String, AttributeValue>` that we can transform to either `PutRequest` or `PutItemRequest`. Then we can evolve this element converter to transparently support either API.
To summarise, I am considering if we actually need a `RequestEntryT` to be something like:
```
interface DynamoDbWriteRequest<T> {
Map<String, AttributeValue> getItems(T record);
// And some other method to decide if it is put or delete
}
```
Then later we can support for `getConditionalExpression()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r999300803
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
I would rather avoid shading the SDK. It means that end users cannot update the SDK without building from source, this has caused issues before. For example, there is no way to add support for new AWS regions for older versions of the connector without building from source. On top of this, if you include multiple connectors, you end up with multiple versions of the SDK bloating the size of the jar.
We can think about a build time check as a follow up https://issues.apache.org/jira/browse/FLINK-29688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r996348036
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Am thinking how this will work for users if there are multiple inflight requests with records that have the same PK/SK.
- Seems like it will be a race condition where the first request to complete will take precedence
- If there are failures/partial failures in a batch, the "latest" record in the stream might no longer take precedence. That said, this situation is more likely to occur if there are more records with the same PK/SK, which helps us, because that means eventual consistency will be attained quickly.
In the scenario above, it seems setting `MAX_IN_FLIGHT_REQUESTS` to 1 would be the main mitigation. I wonder if we should set this as default, or at least call this out to users?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997398631
##########
pom.xml:
##########
@@ -0,0 +1,1140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <parent>
+ <groupId>org.apache</groupId>
+ <artifactId>apache</artifactId>
+ <version>20</version>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connectors</artifactId>
+ <version>1.16-SNAPSHOT</version>
+ <name>Flink : Connectors : </name>
Review Comment:
Missed when copied from the parent project. Fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998512587
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
May be we could provide a setting to turn off deduplication on the sink side. That would be on-pair with boto3 clients for dynamodb (overwrite_by_pkeys=['partition_key', 'sort_key'] config). Wdyt?
> In the future, I was thinking we can implement an improvement to use PutItem API, and the DDB Sink can add an additional key-value with eventTime as the value. Whenever an item with older event time attempts to be written, we can just drop the older record.
Yes I agree that would be a nice feature to add. We actually had it planned for [the first sink version](https://github.com/apache/flink/pull/17407/files#diff-ffcfaf3e245b27ca44831c37b908014be944626c1eac3d5acc63fa310345d937) (not based on AsyncSyncBase). There was interface that allows different types of producers (batching/non-batching) to be able to extend for this use case in the future. We can introduce something similar here and plan for adding this feature. For non-batching sink we could also support conditional expressions for individual PutItem/DeleteItem requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997534733
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
I am also thinking, that if we want to really support that we probably have to remove that deduplication that we are doing in case we find a duplicate key, and instead "split" the batch, to make sure we do not lose records with the same primary key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997909454
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Hi,
Duplicated keys only matter when they get to the same batch request, then dynamodb service will fail the request. I took a simple approach to have accumulator to deduplicate entries based on PK/SK in the same request. E.g. if we had 25 entries and 1 entry has duplicated PK/SK we would then write 24 entries in the request.
But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
If the order of the duplicate/update entries matters I think batch sink can not be used anyway, as the order is mixed up by partial write failures/full retries and parallelism in the sink. Then it is up to user to dedup before the sink.
I may be misunderstood the question. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000808803
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
+ // here, as we have at most 25 elements per batch
+ return UUID.randomUUID().toString();
+ } else {
+ Map<String, AttributeValue> requestItems = getRequestItems(request);
+
+ StringBuilder builder = new StringBuilder();
+ for (String keyName : pKeys) {
+ AttributeValue value = requestItems.get(keyName);
+
+ if (value == null) {
+ throw new InvalidRequestException(
+ "Request " + request.toString() + " does not contain pKey " + keyName);
+ }
+
+ builder.append(getKeyValue(value));
Review Comment:
very good point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1001103974
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, WriteRequest> {
Review Comment:
@nirtsruya sorry I might not have explained my idea well. I actually meant that here we would use your new type:
- `class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {`
Now we can modify `DynamoDbWriteRequest` later to support the conditional fields for non-batch mode. Inside `submitRequestEntries` we would convert `DynamoDbWriteRequest` into `WriteRequest` for batch or `PutItemRequest` for non batch. We would need to also convert `WriteRequest` back into `DynamoDbWriteRequest` for failed records (need to check we have all the required info).
There will be a slight performance hit when we retry, as records are converted multiple times. But this allows us to evolve the connector to support batch/non-batch without having to duplicate all the classes as you explained
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#issuecomment-1243981647
Please rebase on `main` after this is merged https://github.com/apache/flink-connector-dynamodb/pull/2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r966428003
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
If we revert the change, we must ensure that if the dynamo client classes change over time we update our de/serialiser. Otherwise we might get in the situation where user `ElementConverter` generates classes that are not supported by our serialiser. The impact would be that we lose/break attributes when restoring from checkpoints. We can mitigate this by calling it out in the docs/PR template as something to check when updating the SDK.
> I think it may be better to allow the user to define the element converter and not do any translation ourselves (like we had it before this PR
Given we cannot escape these wrappers, I agree with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1009241459
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequestType.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Represents a DynamoDb Write Request type. The following types are currently supported
+ *
+ * <ul>
+ * <li>PUT - Put Request
+ * <li>DELETE - Delete Request
+ * </ul>
+ */
+@PublicEvolving
+public enum DynamoDbWriteRequestType {
+
+ // Note: Enums have no stable hash code across different JVMs, use toByteValue() for
+ // this purpose.
+ PUT((byte) 0),
+ DELETE((byte) 1);
Review Comment:
I think the only concern was keeping order - if order changes for different reasons, the ordinal changes as well - if it is not a concern, we can use the ordinal.
I sawit in the implementation for [Row Kind](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/types/RowKind.java#L106) you linked and assumed it was the reason
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1014057263
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.aws.util.AWSAsyncSinkUtil;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.connector.base.sink.throwable.FatalExceptionClassifier;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getInvalidCredentialsExceptionClassifier;
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getSdkClientMisconfiguredExceptionClassifier;
+import static org.apache.flink.connector.base.sink.writer.AsyncSinkFatalExceptionClassifiers.getInterruptedExceptionClassifier;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ private static final FatalExceptionClassifier RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ResourceNotFoundException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to not being able to find the specified resources",
+ err));
+
+ private static final FatalExceptionClassifier CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ConditionalCheckFailedException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to failed conditional check",
+ err));
+
+ /* Validation exceptions are not retryable. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html. */
+ private static final FatalExceptionClassifier VALIDATION_EXCEPTION_CLASSIFIER =
+ new FatalExceptionClassifier(
+ (err) ->
+ err instanceof DynamoDbException
+ && ((DynamoDbException) err)
+ .toBuilder()
+ .awsErrorDetails()
+ .errorCode()
+ .equalsIgnoreCase("ValidationException"),
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception because of DynamoDB request validation",
+ err));
+
+ private static final FatalExceptionClassifier DYNAMODB_FATAL_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.createChain(
+ getInterruptedExceptionClassifier(),
+ getInvalidCredentialsExceptionClassifier(),
+ RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER,
+ CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER,
+ VALIDATION_EXCEPTION_CLASSIFIER,
+ getSdkClientMisconfiguredExceptionClassifier());
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* A counter for the total number of records that were returned by DynamoDB as unprocessed and were retried */
+ private final Counter numRecordsReturnedAsUnprocessedCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ /* The asynchronous http client for the asynchronous DynamoDb client */
+ private final SdkAsyncHttpClient httpClient;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private final List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPartitionKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.numRecordsReturnedAsUnprocessedCounter =
+ metrics.counter("numRecordsReturnedAsUnprocessedCounter");
+ this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
+ this.client = buildClient(dynamoDbClientProperties, httpClient);
+ }
+
+ private DynamoDbAsyncClient buildClient(
+ Properties dynamoDbClientProperties, SdkAsyncHttpClient httpClient) {
+ AWSGeneralUtil.validateAwsCredentials(dynamoDbClientProperties);
+
+ return AWSAsyncSinkUtil.createAwsAsyncClient(
+ dynamoDbClientProperties,
+ httpClient,
+ DynamoDbAsyncClient.builder(),
+ DynamoDbConfigConstants.BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT,
+ DynamoDbConfigConstants.DYNAMODB_CLIENT_USER_AGENT_PREFIX);
+ }
Review Comment:
with the import
```
import org.apache.flink.connector.aws.config.AWSConfigConstants;
```
Of course
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.aws.util.AWSAsyncSinkUtil;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.connector.base.sink.throwable.FatalExceptionClassifier;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getInvalidCredentialsExceptionClassifier;
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getSdkClientMisconfiguredExceptionClassifier;
+import static org.apache.flink.connector.base.sink.writer.AsyncSinkFatalExceptionClassifiers.getInterruptedExceptionClassifier;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ private static final FatalExceptionClassifier RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ResourceNotFoundException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to not being able to find the specified resources",
+ err));
+
+ private static final FatalExceptionClassifier CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ConditionalCheckFailedException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to failed conditional check",
+ err));
+
+ /* Validation exceptions are not retryable. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html. */
+ private static final FatalExceptionClassifier VALIDATION_EXCEPTION_CLASSIFIER =
+ new FatalExceptionClassifier(
+ (err) ->
+ err instanceof DynamoDbException
+ && ((DynamoDbException) err)
+ .toBuilder()
+ .awsErrorDetails()
+ .errorCode()
+ .equalsIgnoreCase("ValidationException"),
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception because of DynamoDB request validation",
+ err));
+
+ private static final FatalExceptionClassifier DYNAMODB_FATAL_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.createChain(
+ getInterruptedExceptionClassifier(),
+ getInvalidCredentialsExceptionClassifier(),
+ RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER,
+ CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER,
+ VALIDATION_EXCEPTION_CLASSIFIER,
+ getSdkClientMisconfiguredExceptionClassifier());
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* A counter for the total number of records that were returned by DynamoDB as unprocessed and were retried */
+ private final Counter numRecordsReturnedAsUnprocessedCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ /* The asynchronous http client for the asynchronous DynamoDb client */
+ private final SdkAsyncHttpClient httpClient;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private final List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPartitionKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.numRecordsReturnedAsUnprocessedCounter =
+ metrics.counter("numRecordsReturnedAsUnprocessedCounter");
+ this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
+ this.client = buildClient(dynamoDbClientProperties, httpClient);
+ }
+
+ private DynamoDbAsyncClient buildClient(
+ Properties dynamoDbClientProperties, SdkAsyncHttpClient httpClient) {
+ AWSGeneralUtil.validateAwsCredentials(dynamoDbClientProperties);
+
+ return AWSAsyncSinkUtil.createAwsAsyncClient(
+ dynamoDbClientProperties,
+ httpClient,
+ DynamoDbAsyncClient.builder(),
+ DynamoDbConfigConstants.BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT,
+ DynamoDbConfigConstants.DYNAMODB_CLIENT_USER_AGENT_PREFIX);
+ }
Review Comment:
with the import, of course!
```
import org.apache.flink.connector.aws.config.AWSConfigConstants;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-aws] dannycranmer commented on pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on PR #1:
URL: https://github.com/apache/flink-connector-aws/pull/1#issuecomment-1304151007
As discussed offline. The build takes ~9 minutes to run due to excessive use of integration tests. We will follow up to migrate these tests to unit tests, and generally improve coverage ahead of the release:
- https://issues.apache.org/jira/browse/FLINK-29895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1012406782
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.aws.util.AWSAsyncSinkUtil;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.connector.base.sink.throwable.FatalExceptionClassifier;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getInvalidCredentialsExceptionClassifier;
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getSdkClientMisconfiguredExceptionClassifier;
+import static org.apache.flink.connector.base.sink.writer.AsyncSinkFatalExceptionClassifiers.getInterruptedExceptionClassifier;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ private static final FatalExceptionClassifier RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ResourceNotFoundException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to not being able to find the specified resources",
+ err));
+
+ private static final FatalExceptionClassifier CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ConditionalCheckFailedException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to failed conditional check",
+ err));
+
+ /**
+ * Validation exceptions are not retryable. See
+ * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html.
+ */
+ private static final FatalExceptionClassifier VALIDATION_EXCEPTION_CLASSIFIER =
+ new FatalExceptionClassifier(
+ (err) ->
+ err instanceof DynamoDbException
+ && ((DynamoDbException) err)
+ .toBuilder()
+ .awsErrorDetails()
+ .errorCode()
+ .equalsIgnoreCase("ValidationException"),
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception because of DynamoDB request validation",
+ err));
+
+ private static final FatalExceptionClassifier DYNAMODB_FATAL_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.createChain(
+ getInterruptedExceptionClassifier(),
+ getInvalidCredentialsExceptionClassifier(),
+ RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER,
+ CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER,
+ VALIDATION_EXCEPTION_CLASSIFIER,
+ getSdkClientMisconfiguredExceptionClassifier());
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* A counter for the total number of records that were returned by DynamoDB as unprocessed*/
+ private final Counter numRecordsReturnedAsUnprocessedCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ /* The asynchronous http client for the asynchronous DynamoDb client */
+ private final SdkAsyncHttpClient httpClient;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private final List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPartitionKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.numRecordsReturnedAsUnprocessedCounter =
+ metrics.counter("numRecordsReturnedAsUnprocessedCounter");
+ this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
+ this.client =
+ AWSAsyncSinkUtil.createAwsAsyncClient(
+ dynamoDbClientProperties,
+ httpClient,
+ DynamoDbAsyncClient.builder(),
+ DynamoDbConfigConstants.BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT,
+ DynamoDbConfigConstants.DYNAMODB_CLIENT_USER_AGENT_PREFIX);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ List<WriteRequest> items = new ArrayList<>();
+
+ if (CollectionUtil.isNullOrEmpty(overwriteByPartitionKeys)) {
+ for (DynamoDbWriteRequest request : requestEntries) {
+ items.add(convertToWriteRequest(request));
+ }
+ } else {
+ // deduplication needed
+ Map<String, WriteRequest> container = new HashMap<>();
+ PrimaryKeyBuilder keyBuilder = new PrimaryKeyBuilder(overwriteByPartitionKeys);
+ for (DynamoDbWriteRequest request : requestEntries) {
+ WriteRequest req = convertToWriteRequest(request);
+ container.put(keyBuilder.build(req), req);
+ }
+ items.addAll(container.values());
+ }
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(ImmutableMap.of(tableName, items))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (!CollectionUtil.isNullOrEmpty(response.unprocessedItems())) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (String tableName : response.unprocessedItems().keySet()) {
+ for (WriteRequest writeRequest : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(convertToDynamoDbWriteRequest(writeRequest));
+ }
+ }
+
+ LOG.warn("DynamoDB Sink failed to persist and will retry {} entries.", unprocessed.size());
+ numRecordsReturnedAsUnprocessedCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist and will retry {} entries.",
+ requestEntries.size(),
+ err);
+ numRecordsSendErrorsCounter.inc(requestEntries.size());
+
+ if (isRetryable(err.getCause())) {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ private boolean isRetryable(Throwable err) {
+ if (!DYNAMODB_FATAL_EXCEPTION_CLASSIFIER.isFatal(err, getFatalExceptionCons())) {
+ return false;
+ }
+ if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getItem().toString().getBytes(StandardCharsets.UTF_8).length;
Review Comment:
This was fixed. I set to 0 and updated builder class to raise exceptions if corresponding parameters were used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997909454
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Hi,
Duplicated keys only matter when they get to the same batch request, then dynamodb service will fail the request. I took a simple approach to have accumulator to deduplicate entries based on PK/SK in the same request. E.g. if we had 25 entries and 1 entry has duplicated PK/SK we would then write 24 entries in the request.
But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
If order of duplicate/update entries matters I think batch sink can not be used anyway, as the order is mixed up by partial/full retries and parallelism in the sink. Then it is up to user to dedup before the sink.
I may be misunderstood the question. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998629483
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
Review Comment:
Thanks. Sorry can not commit the suggestion as I moved files. I'll fix this manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r996321025
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * add elements will be blocked while the number of elements in the buffer is at the maximum
Review Comment:
```suggestion
* <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
* sink will backpressure while the number of elements in the buffer is at the maximum
```
Nit: This wording might be clearer from a Flink user's POV
##########
flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/TableRequestsContainerTest.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/** Tests for {@link TableRequestsContainer}. */
+public class TableRequestsContainerTest {
+ @Test
+ public void testRequestNotDeduplicatedWhenNoTableConfig() {
+ new Scenario()
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .withDescription("number of requests without deduplication configuration")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testDeduplicatedOnCompositeKey() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk", "sk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ // to be deduplicated
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
Review Comment:
Should we also validate that the newer item takes precedence?
##########
flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/util/AWSDynamoDbUtilTest.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.connector.aws.util.TestUtil;
+import org.apache.flink.streaming.connectors.dynamodb.config.AWSDynamoDbConfigConstants;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
+import software.amazon.awssdk.core.client.config.SdkAdvancedClientOption;
+import software.amazon.awssdk.core.retry.RetryPolicy;
+import software.amazon.awssdk.core.retry.backoff.BackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.EqualJitterBackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.FixedDelayBackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.FullJitterBackoffStrategy;
+
+import java.time.Duration;
+import java.util.Properties;
+
+/** Tests for {@link AWSDynamoDbUtil}. */
+public class AWSDynamoDbUtilTest {
+
+ @Test
+ public void testHasRetryConfiguration() {
+ Properties properties =
+ TestUtil.properties(
+ AWSDynamoDbConfigConstants.THROTTLING_BACKOFF_STRATEGY, "FULL_JITTER");
+
+ Assertions.assertThat(AWSDynamoDbUtil.hasRetryConfiguration(properties))
+ .as("Should have retry configuration")
+ .isTrue();
+ }
+
+ @Test
+ public void testConstructsFullJitterBackoffStrategy() {
+ Properties properties = new Properties();
+ properties.put(AWSDynamoDbConfigConstants.BACKOFF_STRATEGY, "FULL_JITTER");
+ properties.put(AWSDynamoDbConfigConstants.FULL_JITTER_BASE_DELAY_MS, "100");
+ properties.put(AWSDynamoDbConfigConstants.FULL_JITTER_MAX_BACKOFF_TIME_MS, "10");
+
+ BackoffStrategy backoffStrategy =
+ AWSDynamoDbUtil.getBackoffStrategy(
+ properties, AWSDynamoDbConfigConstants.BACKOFF_STRATEGY);
+
+ Assertions.assertThat(backoffStrategy).isInstanceOf(FullJitterBackoffStrategy.class);
+ Assertions.assertThat(((FullJitterBackoffStrategy) backoffStrategy).toBuilder().baseDelay())
+ .isEqualTo(Duration.ofMillis(100));
+ Assertions.assertThat(
+ ((FullJitterBackoffStrategy) backoffStrategy).toBuilder().maxBackoffTime())
+ .isEqualTo(Duration.ofMillis(10));
+ }
+
+ @Test
+ public void testConstructsEqualJitterBackoffStrategy() {
+ Properties properties = new Properties();
+ properties.put(AWSDynamoDbConfigConstants.BACKOFF_STRATEGY, "EQUAL_JITTER");
+ properties.put(AWSDynamoDbConfigConstants.EQUAL_JITTER_BASE_DELAY_MS, "10");
+ properties.put(AWSDynamoDbConfigConstants.EQUAL_JITTER_MAX_BACKOFF_TIME_MS, "100");
+
+ BackoffStrategy backoffStrategy =
+ AWSDynamoDbUtil.getBackoffStrategy(
+ properties, AWSDynamoDbConfigConstants.BACKOFF_STRATEGY);
+
+ Assertions.assertThat(backoffStrategy).isInstanceOf(EqualJitterBackoffStrategy.class);
+ Assertions.assertThat(
+ ((EqualJitterBackoffStrategy) backoffStrategy).toBuilder().baseDelay())
+ .isEqualTo(Duration.ofMillis(10));
+ Assertions.assertThat(
+ ((EqualJitterBackoffStrategy) backoffStrategy).toBuilder().maxBackoffTime())
+ .isEqualTo(Duration.ofMillis(100));
+ }
+
+ @Test
+ public void testConstructsFixedDelayBackoffStrategy() {
+ Properties properties = new Properties();
+ properties.put(AWSDynamoDbConfigConstants.THROTTLING_BACKOFF_STRATEGY, "FIXED_DELAY");
+ properties.put(AWSDynamoDbConfigConstants.FIXED_DELAY_BACKOFF_MS, "10");
+
+ BackoffStrategy backoffStrategy =
+ AWSDynamoDbUtil.getBackoffStrategy(
+ properties, AWSDynamoDbConfigConstants.THROTTLING_BACKOFF_STRATEGY);
+
+ Assertions.assertThat(backoffStrategy).isInstanceOf(FixedDelayBackoffStrategy.class);
+ Assertions.assertThat(backoffStrategy)
+ .isEqualTo(FixedDelayBackoffStrategy.create(Duration.ofMillis(10)));
+ }
+
+ @Test
+ public void testConstructsRetryPolicy() {
+ Properties properties = new Properties();
+
+ properties.put(AWSDynamoDbConfigConstants.BACKOFF_STRATEGY, "FIXED_DELAY");
+ properties.put(AWSDynamoDbConfigConstants.THROTTLING_BACKOFF_STRATEGY, "FIXED_DELAY");
+
+ int expectedNumRetries = 100;
+ properties.put(
+ AWSDynamoDbConfigConstants.NUMBER_RETRIES, String.valueOf(expectedNumRetries));
+ int expectedDurationMs = 10;
+ BackoffStrategy expectedStrategy =
+ FixedDelayBackoffStrategy.create(Duration.ofMillis(expectedDurationMs));
+ properties.put(
+ AWSDynamoDbConfigConstants.FIXED_DELAY_BACKOFF_MS,
+ String.valueOf(expectedDurationMs));
+
+ RetryPolicy policy = AWSDynamoDbUtil.getRetryPolicy(properties);
+
+ Assertions.assertThat(policy.toBuilder().numRetries()).isEqualTo(expectedNumRetries);
+ Assertions.assertThat(policy.toBuilder().backoffStrategy()).isEqualTo(expectedStrategy);
+ Assertions.assertThat(policy.toBuilder().throttlingBackoffStrategy())
+ .isEqualTo(expectedStrategy);
+ }
+
+ @Test
+ public void testConstructsOverrideConfigurationWithDefaults() {
+ Properties properties = new Properties();
+
+ ClientOverrideConfiguration configuration =
+ AWSDynamoDbUtil.getOverrideConfiguration(properties);
+
+ Assertions.assertThat(
+ getAdvancedOption(configuration, SdkAdvancedClientOption.USER_AGENT_PREFIX))
+ .isEqualTo(AWSDynamoDbUtil.getFlinkUserAgentPrefix(properties));
+
+ Assertions.assertThat(configuration.retryPolicy().isPresent()).isTrue();
Review Comment:
nit: For a better message when tests fail
```suggestion
Assertions.assertThat(configuration.retryPolicy()).isEmpty();
```
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
Review Comment:
```suggestion
* <pre>{@code
* private static class DummyDynamoDbElementConverter implements ElementConverter<String, DynamoDbWriteRequest> {
*
* @Override
* public DynamoDbWriteRequest apply(String s) {
* final Map<String, AttributeValue> item = new HashMap<>();
* item.put("your-key", AttributeValue.builder().s(s).build());
* return new DynamoDbWriteRequest(
* "your-table-name",
* WriteRequest.builder()
* .putRequest(PutRequest.builder()
* .item(item)
* .build())
* .build()
* );
* }
* }
* DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
* .setElementConverter(new DummyDynamoDbElementConverter())
* .build();
* }</pre>
```
Updating to reflect latest builder code
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
Review Comment:
```suggestion
* <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000}
```
##########
flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/TableRequestsContainerTest.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/** Tests for {@link TableRequestsContainer}. */
+public class TableRequestsContainerTest {
+ @Test
+ public void testRequestNotDeduplicatedWhenNoTableConfig() {
+ new Scenario()
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .withDescription("number of requests without deduplication configuration")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testDeduplicatedOnCompositeKey() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk", "sk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
Review Comment:
nit: `withItem`
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbTablesConfig tablesConfig;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ DynamoDbTablesConfig tablesConfig,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tablesConfig = tablesConfig;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(tablesConfig);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(container.getRequestItems())
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (String tableName : response.unprocessedItems().keySet()) {
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(tableName, request));
+ }
+ }
+
+ LOG.warn("DynamoDB Sink failed to persist {} entries", unprocessed.size());
Review Comment:
Nit: Might help to also mention these entries will be retried. (same below for `handleFullyFailedRequest()`
##########
flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/TableRequestsContainerTest.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/** Tests for {@link TableRequestsContainer}. */
+public class TableRequestsContainerTest {
+ @Test
+ public void testRequestNotDeduplicatedWhenNoTableConfig() {
+ new Scenario()
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .withDescription("number of requests without deduplication configuration")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testDeduplicatedOnCompositeKey() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk", "sk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ // to be deduplicated
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("tableOne", ImmutableMap.of("pk", "2", "sk", "1"))
+ .withDescription("total number of requests after deduplication")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testDeduplicatedOnPartitionKey() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ // to be deduplicated
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("tableOne", ImmutableMap.of("pk", "2", "sk", "1"))
+ .withDescription(
+ "total number of requests after deduplication on partition key only")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testNotDeduplicatedWhenOnDifferentTable() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk", "sk")
+ .withTableConfig("tableTwo", "pk", "sk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("tableOne", ImmutableMap.of("pk", "2", "sk", "1"))
+ .witItem("tableTwo", ImmutableMap.of("pk", "1", "sk", "1"))
+ .withDescription(
+ "requests should not be deduplecated if belong to different tables")
Review Comment:
```suggestion
"requests should not be deduplicated if belong to different tables")
```
##########
flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/TableRequestsContainerTest.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/** Tests for {@link TableRequestsContainer}. */
+public class TableRequestsContainerTest {
+ @Test
+ public void testRequestNotDeduplicatedWhenNoTableConfig() {
+ new Scenario()
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .withDescription("number of requests without deduplication configuration")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testDeduplicatedOnCompositeKey() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk", "sk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ // to be deduplicated
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("tableOne", ImmutableMap.of("pk", "2", "sk", "1"))
+ .withDescription("total number of requests after deduplication")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testDeduplicatedOnPartitionKey() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ // to be deduplicated
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("tableOne", ImmutableMap.of("pk", "2", "sk", "1"))
+ .withDescription(
+ "total number of requests after deduplication on partition key only")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testNotDeduplicatedWhenOnDifferentTable() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk", "sk")
+ .withTableConfig("tableTwo", "pk", "sk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("tableOne", ImmutableMap.of("pk", "2", "sk", "1"))
+ .witItem("tableTwo", ImmutableMap.of("pk", "1", "sk", "1"))
+ .withDescription(
+ "requests should not be deduplecated if belong to different tables")
+ .withExpectedNumberOfItems(3)
+ .runScenario();
+ }
+
+ private class Scenario {
Review Comment:
Liking this Scenario class! Tests are super easy to read
##########
flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/key/PrimaryKeyTest.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink.key;
+
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+
+/** Unit tests for {@link PrimaryKey}. */
+public class PrimaryKeyTest {
+
+ private static final String PARTITION_KEY_NAME = "part_key_name";
+ private static final String SORT_KEY_NAME = "sort_key_name";
+
+ private ImmutableMap<String, AttributeValue> createItemValues() {
+ return ImmutableMap.of(
+ PARTITION_KEY_NAME,
+ AttributeValue.builder()
+ .s("123")
+ .n("456")
+ .b(SdkBytes.fromString("789", StandardCharsets.UTF_8))
+ .build(),
+ SORT_KEY_NAME,
+ AttributeValue.builder().s("101112").build(),
+ "some_item",
+ AttributeValue.builder().bool(false).build());
+ }
+
+ public WriteRequest createPutItemRequest(Map<String, AttributeValue> itemValues) {
+ return WriteRequest.builder()
+ .putRequest(PutRequest.builder().item(itemValues).build())
+ .build();
+ }
+
+ public WriteRequest createDeleteItemRequest(Map<String, AttributeValue> itemValues) {
+ return WriteRequest.builder()
+ .deleteRequest(DeleteRequest.builder().key(itemValues).build())
+ .build();
+ }
+
+ @Test
+ public void testPartitionKeysOfTwoDifferentRequestsEqual() {
Review Comment:
should we also test the case where `config` is `null`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r996989759
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Very good point, I was thinking about it, I think we should definately add a comment about this.
We need to be specific here, as if we are using parallelism, MAX_IN_FLIGHT_REQUESTS alone will not help and we need to ensure key and ordering from the source as well
But in general, I think we should document that the sink has this "problem"
Maybe in the future we can allow to use single PutItem requests and allow expressions as you mentioned in another comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#issuecomment-1282890400
> Question for general discussion: Do we have any thoughts on using conditional expressions? I note that the `BatchWriteItem` API we are using doesn't support this. Maybe we can create a followup JIRA to track this feature.
I had considered the same. I see that conditional write is supported on the PutItem, but not the Batch api. Based on this, and the discussion above about supporting non-batch mode, I would vote to proceed without it, and add later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000803309
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination table using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * sink will backpressure while the number of elements in the buffer is at the maximum
+ * <li>{@code maxBatchSizeInBytes}: the maximum size of a batch of entries that may be written to
+ * DynamoDb measured in bytes
+ * <li>{@code maxTimeInBufferMS}: the maximum amount of time an entry is allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be flushed immediately
+ * <li>{@code maxRecordSizeInBytes}: the maximum size of a record the sink will accept into the
+ * buffer, a record of size larger than this will be rejected when passed to the sink
+ * <li>{@code failOnError}: when an exception is encountered while persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * <li>{@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in the same batch request.
+ * Only the latest record with the same combination of key attributes is preserved in the
+ * request.
+ * </ul>
+ *
+ * <p>Please see the writer implementation in {@link DynamoDbSinkWriter}
+ *
+ * @param <InputT> Type of the elements handled by this sink
+ */
+@PublicEvolving
+public class DynamoDbSink<InputT> extends AsyncSinkBase<InputT, DynamoDbWriteRequest> {
+
+ private final Properties dynamoDbClientProperties;
+ private final boolean failOnError;
+ private final String tableName;
+ private List<String> overwriteByPKeys;
+
+ protected DynamoDbSink(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
Review Comment:
this one is not necessary to provide, it will be empty list by default which would mean there will be no deduplication
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1001870394
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbElementConverter implements ElementConverter<String, DynamoDbWriteRequest> {
+ *
+ * @Override
+ * public DynamoDbWriteRequest apply(String s) {
+ * final Map<String, AttributeValue> item = new HashMap<>();
+ * item.put("your-key", AttributeValue.builder().s(s).build());
+ * return new DynamoDbWriteRequest(
+ * WriteRequest.builder()
+ * .putRequest(PutRequest.builder()
+ * .item(item)
+ * .build())
+ * .build()
+ * );
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setElementConverter(new DummyDynamoDbElementConverter())
+ * .setDestinationTableName("your-table-name")
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code destinationTableName} destination table for the sink
+ * <li>{@code overwriteByPKeys} will be empty meaning no records deduplication will be performed
Review Comment:
This comment is outdated
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbElementConverter implements ElementConverter<String, DynamoDbWriteRequest> {
+ *
+ * @Override
+ * public DynamoDbWriteRequest apply(String s) {
+ * final Map<String, AttributeValue> item = new HashMap<>();
+ * item.put("your-key", AttributeValue.builder().s(s).build());
+ * return new DynamoDbWriteRequest(
+ * WriteRequest.builder()
+ * .putRequest(PutRequest.builder()
+ * .item(item)
+ * .build())
+ * .build()
+ * );
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setElementConverter(new DummyDynamoDbElementConverter())
+ * .setDestinationTableName("your-table-name")
+ * .build();
Review Comment:
This comment is now outdated
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequest.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Represents a single Write Request to DynamoDb. Contains the item to be written as well as the
+ * type of the Write Request (PUT/DELETE)
+ */
+@PublicEvolving
+public class DynamoDbWriteRequest implements Serializable {
Review Comment:
https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization
> Serializable classes must define a Serial Version UID:
> private static final long serialVersionUID = 1L;
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequest.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Represents a single Write Request to DynamoDb. Contains the item to be written as well as the
+ * type of the Write Request (PUT/DELETE)
+ */
+@PublicEvolving
+public class DynamoDbWriteRequest implements Serializable {
+
+ private final Map<String, AttributeValue> item;
+ private final DynamoDbWriteRequestType type;
+
+ private DynamoDbWriteRequest(Map<String, AttributeValue> item, DynamoDbWriteRequestType type) {
+ this.item = item;
+ this.type = type;
+ }
+
+ public Map<String, AttributeValue> getItem() {
+ return item;
+ }
+
+ public DynamoDbWriteRequestType getType() {
+ return type;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ DynamoDbWriteRequest that = (DynamoDbWriteRequest) o;
+ return item.equals(that.item) && type == that.type;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(item, type);
+ }
Review Comment:
Why is this needed?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/TableRequestsContainer.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Container to accumulate requests in batches per table. De-duplicates batch request entities as
+ * per PrimaryKey definition. DynamoDB Batch API rejects the whole batch request if the request
+ * contains at least two items with identical hash and range keys (which essentially is two put
+ * operations).
+ */
+@Internal
+class TableRequestsContainer {
+
+ private final Map<String, WriteRequest> container;
+
+ private List<String> overwriteByPKeys;
Review Comment:
Please update field name to match the other one `overwriteByPartitionKeys`
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
Review Comment:
Can we update name to `overwriteByPartitionKeys` to match the other classes?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ for (DynamoDbWriteRequest requestEntry : requestEntries) {
+ container.put(convertToWriteRequest(requestEntry));
+ }
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (!CollectionUtil.isNullOrEmpty(response.unprocessedItems())) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (String tableName : response.unprocessedItems().keySet()) {
+ for (WriteRequest writeRequest : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(convertToDynamoDbWriteRequest(writeRequest));
+ }
+ }
+
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ unprocessed.size());
+ numRecordsSendErrorsCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ requestEntries.size(),
+ err);
+ numRecordsSendErrorsCounter.inc(requestEntries.size());
+
+ if (DynamoDbExceptionUtils.isNotRetryableException(err.getCause())) {
+ getFatalExceptionCons()
+ .accept(
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception", err));
+ } else if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ } else {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getItem().toString().getBytes(StandardCharsets.UTF_8).length;
+ }
+
+ private WriteRequest convertToWriteRequest(DynamoDbWriteRequest dynamoDbWriteRequest) {
+ if (dynamoDbWriteRequest.getType() == DynamoDbWriteRequestType.PUT) {
+ return WriteRequest.builder()
+ .putRequest(PutRequest.builder().item(dynamoDbWriteRequest.getItem()).build())
+ .build();
+ } else if (dynamoDbWriteRequest.getType() == DynamoDbWriteRequestType.DELETE) {
+ return WriteRequest.builder()
+ .deleteRequest(
+ DeleteRequest.builder().key(dynamoDbWriteRequest.getItem()).build())
+ .build();
+ } else {
+ throw new IllegalArgumentException("");
+ }
+ }
+
+ private DynamoDbWriteRequest convertToDynamoDbWriteRequest(WriteRequest writeRequest) {
+ if (writeRequest.putRequest() != null) {
+ return DynamoDbWriteRequest.build()
+ .setItem(writeRequest.putRequest().item())
+ .setType(DynamoDbWriteRequestType.PUT)
+ .build();
+ } else if (writeRequest.deleteRequest() != null) {
+ return DynamoDbWriteRequest.build()
+ .setItem(writeRequest.deleteRequest().key())
+ .setType(DynamoDbWriteRequestType.DELETE)
+ .build();
+ } else {
+ throw new IllegalArgumentException("");
Review Comment:
Same as above
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ for (DynamoDbWriteRequest requestEntry : requestEntries) {
+ container.put(convertToWriteRequest(requestEntry));
+ }
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
Review Comment:
As an optimization, we could completely skip `PrimaryKeyBuilder.build` when `CollectionUtil.isNullOrEmpty(overwriteByPKeys)`. This would remove the UUID generation and map overhead. Something like this:
```
final List<WriteRequest> writeRequests;
if (CollectionUtil.isNullOrEmpty(overwriteByPKeys)) {
writeRequests = new ArrayList<>();
for (DynamoDbWriteRequest requestEntry : requestEntries) {
writeRequests.put(convertToWriteRequest(requestEntry));
}
} else {
TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
for (DynamoDbWriteRequest requestEntry : requestEntries) {
container.put(convertToWriteRequest(requestEntry));
}
writeRequests = container.getRequestItems();
}
CompletableFuture<BatchWriteItemResponse> future =
client.batchWriteItem(
BatchWriteItemRequest.builder()
.requestItems(ImmutableMap.of(tableName, writeRequests))
.build());
```
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ for (DynamoDbWriteRequest requestEntry : requestEntries) {
+ container.put(convertToWriteRequest(requestEntry));
+ }
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (!CollectionUtil.isNullOrEmpty(response.unprocessedItems())) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (String tableName : response.unprocessedItems().keySet()) {
+ for (WriteRequest writeRequest : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(convertToDynamoDbWriteRequest(writeRequest));
+ }
+ }
+
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ unprocessed.size());
+ numRecordsSendErrorsCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ requestEntries.size(),
+ err);
+ numRecordsSendErrorsCounter.inc(requestEntries.size());
+
+ if (DynamoDbExceptionUtils.isNotRetryableException(err.getCause())) {
+ getFatalExceptionCons()
+ .accept(
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception", err));
+ } else if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ } else {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getItem().toString().getBytes(StandardCharsets.UTF_8).length;
+ }
+
+ private WriteRequest convertToWriteRequest(DynamoDbWriteRequest dynamoDbWriteRequest) {
+ if (dynamoDbWriteRequest.getType() == DynamoDbWriteRequestType.PUT) {
+ return WriteRequest.builder()
+ .putRequest(PutRequest.builder().item(dynamoDbWriteRequest.getItem()).build())
+ .build();
+ } else if (dynamoDbWriteRequest.getType() == DynamoDbWriteRequestType.DELETE) {
+ return WriteRequest.builder()
+ .deleteRequest(
+ DeleteRequest.builder().key(dynamoDbWriteRequest.getItem()).build())
+ .build();
+ } else {
+ throw new IllegalArgumentException("");
Review Comment:
Can we add a message to the exception? Given this should not happen we can quite explicitly say the `convertToWriteRequest` method must need updating
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003526661
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(request));
+ }
+
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ unprocessed.size());
+ numRecordsOutErrorsCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ requestEntries.size(),
+ err);
+ numRecordsOutErrorsCounter.inc(requestEntries.size());
+
+ if (DynamoDbExceptionUtils.isNotRetryableException(err.getCause())) {
+ getFatalExceptionCons()
+ .accept(
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception", err));
+ } else if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ } else {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getWriteRequest().toString().getBytes(StandardCharsets.UTF_8).length;
Review Comment:
Hi @dannycranmer,
It returns write request serialised into string with all attribute types (pretty much like dynamodb json format). Something like
WriteRequest(PutRequest=PutRequest(Item={sk=AttributeValue(S=1), pk=AttributeValue(S=1), ...}))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r999086629
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
@dannycranmer makes sense, I can go ahead with implementing that if you agree @YuriGusev
regarding @sthm comment on serialization, do we want to shade the dynamodb sdk? is it something the community was considering?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r965983155
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
It looks like we have created wrappers around the AWS SDK DynamoDB model within the sink. Can you please explain your reasoning behind this approach? The idea behind hiding the ElementConverter is to 1/ hide the internal implementation and 2/ enable sink to manage de/serialisation. However I am struggling to see the benefits here since the 2x models are very similar. If we will copy the underlying model it might be best to use that model, since users might already be creating these objects.
Did you consider using a more generic `ElementConverter`, like the [DynamoDBMapper](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.html)? I am not familiar with this so not sure if there are performance/feature limitations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#issuecomment-1245960254
Thanks for the review @dannycranmer, I'm currently away from home, bit I hope to start working on the fixes next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997544803
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
@YuriGusev I think it makes sense, what do you think? should we split batches instead of deduplication?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997399953
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Yes you're right. Users will also need to ensure that a `keyBy` is done before the sink such that all records with same SK/PK goes to the same subtask.
Yep, agreed that we can set the default, and create a JIRA for using single PutItem requests with conditional expressions in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997909454
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Hi,
Duplicated keys only matter when they get to the same batch request, then dynamodb service will fail the request. I took a simple approach to have accumulator to deduplicate entries based on PK/SK in the same request. E.g. if we had 25 entries and 1 entry has duplicated PK/SK we would then write 24 entries in the request.
But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
If order of the duplicate/update entries matters I think batch sink can not be used anyway, as the order is mixed up by partial write failures/full retries and parallelism in the sink. Then it is up to user to dedup before the sink.
I may be misunderstood the question. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997909454
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Hi,
Duplicated keys only matter when they get to the same batch request, then dynamodb service will fail the request. I took a simple approach to have accumulator to deduplicate entries based on PK/SK in the same request. E.g. if we had 25 entries and 1 entry has duplicated PK/SK we would then write 24 entries in the request.
But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
If order of the duplicate/update entries matters I think batch sink can not be used anyway, as the order is mixed up by partial/full retries and parallelism in the sink. Then it is up to user to dedup before the sink.
I may be misunderstood the question. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998653397
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Yes this is right if there is no configuration it won't deduplicate anything. It's been a while I forgot I added test for this :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r999919824
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (String tableName : response.unprocessedItems().keySet()) {
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(request));
+ }
+ }
Review Comment:
Should we just directly get the current table name and remove the outer for loop?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb table configuration provided. Using UUID should be save
+ // here, as we have at most 25 elements per batch
Review Comment:
```suggestion
// fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
// here, as we have at most 25 elements per batch
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000808138
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbElementConverter implements ElementConverter<String, DynamoDbWriteRequest> {
+ *
+ * @Override
+ * public DynamoDbWriteRequest apply(String s) {
+ * final Map<String, AttributeValue> item = new HashMap<>();
+ * item.put("your-key", AttributeValue.builder().s(s).build());
+ * return new DynamoDbWriteRequest(
+ * WriteRequest.builder()
+ * .putRequest(PutRequest.builder()
+ * .item(item)
+ * .build())
+ * .build()
+ * );
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setElementConverter(new DummyDynamoDbElementConverter())
+ * .setDestinationTableName("your-table-name")
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code destinationTableName} destination table for the sink
+ * <li>{@code overwriteByPKeys} will be empty meaning no records deduplication will be performed
+ * by the batch sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
+ private static final int DEFAULT_MAX_BUFFERED_REQUESTS = 10000;
+ private static final long DEFAULT_MAX_BATCH_SIZE_IN_B = 16 * 1000 * 1000;
+ private static final long DEFAULT_MAX_TIME_IN_BUFFER_MS = 5000;
+ private static final long DEFAULT_MAX_RECORD_SIZE_IN_B = 400 * 1000;
+ private static final boolean DEFAULT_FAIL_ON_ERROR = false;
+
+ private boolean failOnError;
+ private Properties dynamodbClientProperties;
+
+ private ElementConverter<InputT, DynamoDbWriteRequest> elementConverter;
+ private String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkBuilder<InputT> setDynamoDbProperties(Properties properties) {
+ this.dynamodbClientProperties = properties;
+ return this;
+ }
+
+ public DynamoDbSinkBuilder<InputT> setElementConverter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter) {
+ this.elementConverter = elementConverter;
+ return this;
+ }
+
+ /** Destination DynamoDB table name for the sink to write to. */
+ public DynamoDbSinkBuilder<InputT> setDestinationTableName(String tableName) {
+ this.tableName = tableName;
+ return this;
+ }
+
+ /**
+ * @param overwriteByPKeys provide partition key and (optionally) sort key name if you want to
+ * bypass no duplication limitation of single batch write request. Batching DynamoDB sink
+ * will drop request items in the buffer if their primary keys(composite) values are the
+ * same as newly added one.
Review Comment:
We had it as a separate DTO, I changed it [here](https://github.com/apache/flink-connector-dynamodb/pull/1/commits/b51a075733618e6b9110e5253e9ee5450b4b6e8c#diff-d72f905550d45db880aa1a62971000bd9aeedf89a8f80a881da0e9266377d022). Order/number of keys doesn't matter for the deduplication logic. I can fix the comment to make it more clear, or return previous approach.
I changed this is to match similar approach in the [boto3 dynamodb batch client](https://github.com/boto/boto3/blob/develop/boto3/dynamodb/table.py#L52).
Do you think it make sense to keep this way? Otherwise I'm fine to rollback to the previous version (however we would need to keep nullable field in the builder)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000814379
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequest.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+/**
+ * Represents a single DynamoDb {@link WriteRequest}. TODO remove this class, replace with interface
+ * to support batch and non-batch modes
+ */
+@Internal
+public class DynamoDbWriteRequest implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private final WriteRequest writeRequest;
+
+ public DynamoDbWriteRequest(WriteRequest writeRequest) {
+ this.writeRequest = writeRequest;
+ }
+
+ public WriteRequest getWriteRequest() {
+ return writeRequest;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ DynamoDbWriteRequest that = (DynamoDbWriteRequest) o;
+ return Objects.equals(writeRequest, that.writeRequest);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(writeRequest);
+ }
Review Comment:
I left this class to remove later when we migrate to the interface approach (@nirtsruya) is working on that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003536293
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
Review Comment:
I agree, I'll work on removing retries logic and using standard base class. It was developed on top of base classes when kinesis connector was actively worked on. We can rely on standard retry for now (can pick up FLINK-29683 later to make it configurable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1008763069
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
+ // here, as we have at most 25 elements per batch
+ return UUID.randomUUID().toString();
+ } else {
+ Map<String, AttributeValue> requestItems = getRequestItems(request);
+
+ StringBuilder builder = new StringBuilder();
+ for (String keyName : pKeys) {
+ AttributeValue value = requestItems.get(keyName);
+
+ if (value == null) {
+ throw new InvalidRequestException(
+ "Request " + request.toString() + " does not contain pKey " + keyName);
+ }
+
+ builder.append(getKeyValue(value));
+ }
+
+ return builder.toString();
+ }
+ }
+
+ /**
+ * Returns string value of a partition key attribute. Each primary key attribute must be defined
+ * as type String, Number, or binary as per DynamoDB specification.
+ */
+ private static String getKeyValue(AttributeValue value) {
+ StringBuilder builder = new StringBuilder();
+
+ if (value.n() != null) {
+ builder.append(value.n());
+ }
+
+ if (value.s() != null) {
+ builder.append(value.s());
+ }
+
+ if (value.b() != null) {
+ builder.append(value.b().asUtf8String());
+ }
+
+ return builder.toString();
Review Comment:
I added validation as per DynamoDB specification partition key, and sort key values can not be blank. However, we would only be able to validate that if the deduplication configuration is provided. When not provided, the request will be failed by DynamoDB.
DynamoDB would fail the request if we do not validate for the blank values at all, but I agree it is good to fail fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1013131920
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbSerializationUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbWriteRequest;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbWriteRequestType;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Serialization Utils for DynamoDb {@link AttributeValue}. This class is currently not
+ * serializable, see <a href="https://github.com/aws/aws-sdk-java-v2/issues/3143">open issue</a>
+ */
+@Internal
+public class DynamoDbSerializationUtil {
+
+ public static void serializeWriteRequest(
+ DynamoDbWriteRequest dynamoDbWriteRequest, DataOutputStream out) throws IOException {
+ out.writeByte(dynamoDbWriteRequest.getType().toByteValue());
+ Map<String, AttributeValue> item = dynamoDbWriteRequest.getItem();
+ serializeItem(item, out);
+ }
+
+ public static DynamoDbWriteRequest deserializeWriteRequest(DataInputStream in)
+ throws IOException {
+ int writeRequestType = in.read();
+ DynamoDbWriteRequestType dynamoDbWriteRequestType =
+ DynamoDbWriteRequestType.fromByteValue((byte) writeRequestType);
+ Map<String, AttributeValue> item = deserializeItem(in);
+ return DynamoDbWriteRequest.builder()
+ .setType(dynamoDbWriteRequestType)
+ .setItem(item)
+ .build();
+ }
+
+ private static void serializeItem(Map<String, AttributeValue> item, DataOutputStream out)
+ throws IOException {
+ out.writeInt(item.size());
+ for (Map.Entry<String, AttributeValue> entry : item.entrySet()) {
+ out.writeUTF(entry.getKey());
+ AttributeValue value = entry.getValue();
+ serializeAttributeValue(value, out);
+ }
+ }
+
+ private static void serializeAttributeValue(AttributeValue value, DataOutputStream out)
+ throws IOException {
+ if (value.nul() != null) {
+ out.writeByte(DynamoDbType.NULL.toByteValue());
+ } else if (value.bool() != null) {
+ out.writeByte(DynamoDbType.BOOLEAN.toByteValue());
+ out.writeBoolean(value.bool());
+ } else if (value.s() != null) {
+ out.writeByte(DynamoDbType.STRING.toByteValue());
+ out.writeUTF(value.s());
+ } else if (value.n() != null) {
+ out.writeByte(DynamoDbType.NUMBER.toByteValue());
+ out.writeUTF(value.n());
+ } else if (value.b() != null) {
+ out.writeByte(DynamoDbType.BINARY.toByteValue());
+ out.writeInt(value.b().asByteArrayUnsafe().length);
+ out.write(value.b().asByteArrayUnsafe());
Review Comment:
No, it simply returns a member, but in any case, makes sense to extract
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1014006791
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.aws.util.AWSAsyncSinkUtil;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.connector.base.sink.throwable.FatalExceptionClassifier;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getInvalidCredentialsExceptionClassifier;
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getSdkClientMisconfiguredExceptionClassifier;
+import static org.apache.flink.connector.base.sink.writer.AsyncSinkFatalExceptionClassifiers.getInterruptedExceptionClassifier;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ private static final FatalExceptionClassifier RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ResourceNotFoundException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to not being able to find the specified resources",
+ err));
+
+ private static final FatalExceptionClassifier CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ConditionalCheckFailedException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to failed conditional check",
+ err));
+
+ /* Validation exceptions are not retryable. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html. */
+ private static final FatalExceptionClassifier VALIDATION_EXCEPTION_CLASSIFIER =
+ new FatalExceptionClassifier(
+ (err) ->
+ err instanceof DynamoDbException
+ && ((DynamoDbException) err)
+ .toBuilder()
+ .awsErrorDetails()
+ .errorCode()
+ .equalsIgnoreCase("ValidationException"),
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception because of DynamoDB request validation",
+ err));
+
+ private static final FatalExceptionClassifier DYNAMODB_FATAL_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.createChain(
+ getInterruptedExceptionClassifier(),
+ getInvalidCredentialsExceptionClassifier(),
+ RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER,
+ CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER,
+ VALIDATION_EXCEPTION_CLASSIFIER,
+ getSdkClientMisconfiguredExceptionClassifier());
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* A counter for the total number of records that were returned by DynamoDB as unprocessed and were retried */
+ private final Counter numRecordsReturnedAsUnprocessedCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ /* The asynchronous http client for the asynchronous DynamoDb client */
+ private final SdkAsyncHttpClient httpClient;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private final List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPartitionKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.numRecordsReturnedAsUnprocessedCounter =
+ metrics.counter("numRecordsReturnedAsUnprocessedCounter");
+ this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
Review Comment:
```suggestion
dynamoDbClientProperties.put(AWSConfigConstants.HTTP_PROTOCOL_VERSION, "HTTP1_1");
this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1009328645
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequestType.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Represents a DynamoDb Write Request type. The following types are currently supported
+ *
+ * <ul>
+ * <li>PUT - Put Request
+ * <li>DELETE - Delete Request
+ * </ul>
+ */
+@PublicEvolving
+public enum DynamoDbWriteRequestType {
+
+ // Note: Enums have no stable hash code across different JVMs, use toByteValue() for
+ // this purpose.
+ PUT((byte) 0),
+ DELETE((byte) 1);
Review Comment:
I had linked the RowKind as an option to completely replace the enum we were using. However I am happy to stick with what we have got since they do not line up 100%. I agree that the byte mechanism is immune to ordering errors, but it seems like we could mitigate this with a good unit test. I will leave this decision up to you, I do not mind either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003539179
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
Review Comment:
sorry missed this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1008763069
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
+ // here, as we have at most 25 elements per batch
+ return UUID.randomUUID().toString();
+ } else {
+ Map<String, AttributeValue> requestItems = getRequestItems(request);
+
+ StringBuilder builder = new StringBuilder();
+ for (String keyName : pKeys) {
+ AttributeValue value = requestItems.get(keyName);
+
+ if (value == null) {
+ throw new InvalidRequestException(
+ "Request " + request.toString() + " does not contain pKey " + keyName);
+ }
+
+ builder.append(getKeyValue(value));
+ }
+
+ return builder.toString();
+ }
+ }
+
+ /**
+ * Returns string value of a partition key attribute. Each primary key attribute must be defined
+ * as type String, Number, or binary as per DynamoDB specification.
+ */
+ private static String getKeyValue(AttributeValue value) {
+ StringBuilder builder = new StringBuilder();
+
+ if (value.n() != null) {
+ builder.append(value.n());
+ }
+
+ if (value.s() != null) {
+ builder.append(value.s());
+ }
+
+ if (value.b() != null) {
+ builder.append(value.b().asUtf8String());
+ }
+
+ return builder.toString();
Review Comment:
I added validation as per DynamoDB specification partition key, and sort key values can not be blank. However, we would only be able to validate that if the deduplication configuration is provided. When not provided, the request will be failed by DynamoDB.
DynamoDB would vail request if we do not validate for the blank values at all, but I agree it is good to fail fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1012223894
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.aws.util.AWSAsyncSinkUtil;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.connector.base.sink.throwable.FatalExceptionClassifier;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getInvalidCredentialsExceptionClassifier;
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getSdkClientMisconfiguredExceptionClassifier;
+import static org.apache.flink.connector.base.sink.writer.AsyncSinkFatalExceptionClassifiers.getInterruptedExceptionClassifier;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ private static final FatalExceptionClassifier RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ResourceNotFoundException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to not being able to find the specified resources",
+ err));
+
+ private static final FatalExceptionClassifier CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ConditionalCheckFailedException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to failed conditional check",
+ err));
+
+ /**
+ * Validation exceptions are not retryable. See
+ * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html.
+ */
+ private static final FatalExceptionClassifier VALIDATION_EXCEPTION_CLASSIFIER =
+ new FatalExceptionClassifier(
+ (err) ->
+ err instanceof DynamoDbException
+ && ((DynamoDbException) err)
+ .toBuilder()
+ .awsErrorDetails()
+ .errorCode()
+ .equalsIgnoreCase("ValidationException"),
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception because of DynamoDB request validation",
+ err));
+
+ private static final FatalExceptionClassifier DYNAMODB_FATAL_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.createChain(
+ getInterruptedExceptionClassifier(),
+ getInvalidCredentialsExceptionClassifier(),
+ RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER,
+ CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER,
+ VALIDATION_EXCEPTION_CLASSIFIER,
+ getSdkClientMisconfiguredExceptionClassifier());
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* A counter for the total number of records that were returned by DynamoDB as unprocessed*/
+ private final Counter numRecordsReturnedAsUnprocessedCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ /* The asynchronous http client for the asynchronous DynamoDb client */
+ private final SdkAsyncHttpClient httpClient;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private final List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPartitionKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.numRecordsReturnedAsUnprocessedCounter =
+ metrics.counter("numRecordsReturnedAsUnprocessedCounter");
+ this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
+ this.client =
+ AWSAsyncSinkUtil.createAwsAsyncClient(
+ dynamoDbClientProperties,
+ httpClient,
+ DynamoDbAsyncClient.builder(),
+ DynamoDbConfigConstants.BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT,
+ DynamoDbConfigConstants.DYNAMODB_CLIENT_USER_AGENT_PREFIX);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ List<WriteRequest> items = new ArrayList<>();
+
+ if (CollectionUtil.isNullOrEmpty(overwriteByPartitionKeys)) {
+ for (DynamoDbWriteRequest request : requestEntries) {
+ items.add(convertToWriteRequest(request));
+ }
+ } else {
+ // deduplication needed
+ Map<String, WriteRequest> container = new HashMap<>();
+ PrimaryKeyBuilder keyBuilder = new PrimaryKeyBuilder(overwriteByPartitionKeys);
+ for (DynamoDbWriteRequest request : requestEntries) {
+ WriteRequest req = convertToWriteRequest(request);
+ container.put(keyBuilder.build(req), req);
+ }
+ items.addAll(container.values());
+ }
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(ImmutableMap.of(tableName, items))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (!CollectionUtil.isNullOrEmpty(response.unprocessedItems())) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (String tableName : response.unprocessedItems().keySet()) {
+ for (WriteRequest writeRequest : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(convertToDynamoDbWriteRequest(writeRequest));
+ }
+ }
+
+ LOG.warn("DynamoDB Sink failed to persist and will retry {} entries.", unprocessed.size());
+ numRecordsReturnedAsUnprocessedCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist and will retry {} entries.",
+ requestEntries.size(),
+ err);
+ numRecordsSendErrorsCounter.inc(requestEntries.size());
+
+ if (isRetryable(err.getCause())) {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ private boolean isRetryable(Throwable err) {
+ if (!DYNAMODB_FATAL_EXCEPTION_CLASSIFIER.isFatal(err, getFatalExceptionCons())) {
+ return false;
+ }
+ if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getItem().toString().getBytes(StandardCharsets.UTF_8).length;
Review Comment:
As discussed offline, it is not practical to compute the `sizeInBytes` given that:
- We do not know how the client serializes the request payload
- This will be an expensive operation and we would potentially be serializing each record twice
We agree to:
- Return a dummy constant value here
- Follow up with an improvement to make this flush strategy and validation optional ([FLINK-29854](https://issues.apache.org/jira/browse/FLINK-29854))
The failure modes are already covered:
- Total batch size too large: this will be handled organically by AIMD
- Single `DynamoDbWriteRequest` too large: if we try to send a single request that is too large it will result in a fatal validation exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1013142220
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbSerializationUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbWriteRequest;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbWriteRequestType;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Serialization Utils for DynamoDb {@link AttributeValue}. This class is currently not
+ * serializable, see <a href="https://github.com/aws/aws-sdk-java-v2/issues/3143">open issue</a>
+ */
+@Internal
+public class DynamoDbSerializationUtil {
+
+ public static void serializeWriteRequest(
+ DynamoDbWriteRequest dynamoDbWriteRequest, DataOutputStream out) throws IOException {
+ out.writeByte(dynamoDbWriteRequest.getType().toByteValue());
+ Map<String, AttributeValue> item = dynamoDbWriteRequest.getItem();
+ serializeItem(item, out);
+ }
+
+ public static DynamoDbWriteRequest deserializeWriteRequest(DataInputStream in)
+ throws IOException {
+ int writeRequestType = in.read();
+ DynamoDbWriteRequestType dynamoDbWriteRequestType =
+ DynamoDbWriteRequestType.fromByteValue((byte) writeRequestType);
+ Map<String, AttributeValue> item = deserializeItem(in);
+ return DynamoDbWriteRequest.builder()
+ .setType(dynamoDbWriteRequestType)
+ .setItem(item)
+ .build();
+ }
+
+ private static void serializeItem(Map<String, AttributeValue> item, DataOutputStream out)
+ throws IOException {
+ out.writeInt(item.size());
+ for (Map.Entry<String, AttributeValue> entry : item.entrySet()) {
+ out.writeUTF(entry.getKey());
+ AttributeValue value = entry.getValue();
+ serializeAttributeValue(value, out);
+ }
+ }
+
+ private static void serializeAttributeValue(AttributeValue value, DataOutputStream out)
+ throws IOException {
+ if (value.nul() != null) {
+ out.writeByte(DynamoDbType.NULL.toByteValue());
+ } else if (value.bool() != null) {
+ out.writeByte(DynamoDbType.BOOLEAN.toByteValue());
+ out.writeBoolean(value.bool());
+ } else if (value.s() != null) {
+ out.writeByte(DynamoDbType.STRING.toByteValue());
+ out.writeUTF(value.s());
+ } else if (value.n() != null) {
+ out.writeByte(DynamoDbType.NUMBER.toByteValue());
+ out.writeUTF(value.n());
+ } else if (value.b() != null) {
+ out.writeByte(DynamoDbType.BINARY.toByteValue());
+ out.writeInt(value.b().asByteArrayUnsafe().length);
+ out.write(value.b().asByteArrayUnsafe());
Review Comment:
In this case, I do not mind either way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1013066455
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidRequestException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB request was invalid. */
+public class InvalidRequestException extends RuntimeException {
Review Comment:
Missing `@PublicEvolving`
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbSerializationUtil.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbWriteRequest;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbWriteRequestType;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Serialization Utils for DynamoDb {@link AttributeValue}. This class is currently not
+ * serializable, see <a href="https://github.com/aws/aws-sdk-java-v2/issues/3143">open issue</a>
+ */
+@Internal
+public class DynamoDbSerializationUtil {
+
+ public static void serializeWriteRequest(
+ DynamoDbWriteRequest dynamoDbWriteRequest, DataOutputStream out) throws IOException {
+ out.writeByte(dynamoDbWriteRequest.getType().toByteValue());
+ Map<String, AttributeValue> item = dynamoDbWriteRequest.getItem();
+ serializeItem(item, out);
+ }
+
+ public static DynamoDbWriteRequest deserializeWriteRequest(DataInputStream in)
+ throws IOException {
+ int writeRequestType = in.read();
+ DynamoDbWriteRequestType dynamoDbWriteRequestType =
+ DynamoDbWriteRequestType.fromByteValue((byte) writeRequestType);
+ Map<String, AttributeValue> item = deserializeItem(in);
+ return DynamoDbWriteRequest.builder()
+ .setType(dynamoDbWriteRequestType)
+ .setItem(item)
+ .build();
+ }
+
+ private static void serializeItem(Map<String, AttributeValue> item, DataOutputStream out)
+ throws IOException {
+ out.writeInt(item.size());
+ for (Map.Entry<String, AttributeValue> entry : item.entrySet()) {
+ out.writeUTF(entry.getKey());
+ AttributeValue value = entry.getValue();
+ serializeAttributeValue(value, out);
+ }
+ }
+
+ private static void serializeAttributeValue(AttributeValue value, DataOutputStream out)
+ throws IOException {
+ if (value.nul() != null) {
+ out.writeByte(DynamoDbType.NULL.toByteValue());
+ } else if (value.bool() != null) {
+ out.writeByte(DynamoDbType.BOOLEAN.toByteValue());
+ out.writeBoolean(value.bool());
+ } else if (value.s() != null) {
+ out.writeByte(DynamoDbType.STRING.toByteValue());
+ out.writeUTF(value.s());
+ } else if (value.n() != null) {
+ out.writeByte(DynamoDbType.NUMBER.toByteValue());
+ out.writeUTF(value.n());
+ } else if (value.b() != null) {
+ out.writeByte(DynamoDbType.BINARY.toByteValue());
+ out.writeInt(value.b().asByteArrayUnsafe().length);
+ out.write(value.b().asByteArrayUnsafe());
Review Comment:
Does `asByteArrayUnsafe` result in a new byte array getting created? If so, can we extract to reduce this happening twice
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkException.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown when DynamoDb sink failed to write data. */
+public class DynamoDbSinkException extends RuntimeException {
Review Comment:
I see the annotations were recently removed, why was this? https://github.com/apache/flink-connector-dynamodb/pull/1/commits/598373ace67c96e1e7fb466f60129f8475738bf9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1013058794
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbElementConverter
+ * implements ElementConverter<String, DynamoDbWriteRequest> {
+ *
+ * @Override
+ * public DynamoDbWriteRequest apply(String s, SinkWriter.Context context) {
+ * final Map<String, AttributeValue> item = new HashMap<>();
+ * item.put("your-key", AttributeValue.builder().s(s).build());
+ * return DynamoDbWriteRequest.builder()
+ * .setItem(item)
+ * .setType(DynamoDbWriteRequestType.PUT)
+ * .build();
+ * }
+ * }
+ *
+ * DynamoDbSink<String> dynamoDbSink =
+ * DynamoDbSink.<String>builder()
+ * .setElementConverter(new DummyDynamoDbElementConverter())
+ * .setDestinationTableName("your-table-name")
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} setting is not supported by this implementation
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} setting is not supported by this implementation
+ * <li>{@code failOnError} will be false
+ * <li>{@code destinationTableName} destination table for the sink
+ * <li>{@code overwriteByPartitionKeys} will be empty meaning no records deduplication will be
+ * performed by the batch sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
+ private static final int DEFAULT_MAX_BUFFERED_REQUESTS = 10000;
+
+ // Max record size in bytes and max batch size in bytes are not supported by this implementation
+ private static final long DEFAULT_MAX_RECORD_SIZE_IN_B = 400 * 1000;
+ private static final long DEFAULT_MAX_BATCH_SIZE_IN_B = 16 * 1000 * 1000;
+
+ private static final long DEFAULT_MAX_TIME_IN_BUFFER_MS = 5000;
+ private static final boolean DEFAULT_FAIL_ON_ERROR = false;
+
+ private boolean failOnError;
+ private Properties dynamodbClientProperties;
+
+ private ElementConverter<InputT, DynamoDbWriteRequest> elementConverter;
+ private String tableName;
+
+ private List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkBuilder<InputT> setDynamoDbProperties(Properties properties) {
+ this.dynamodbClientProperties = properties;
+ return this;
+ }
+
+ public DynamoDbSinkBuilder<InputT> setElementConverter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter) {
+ this.elementConverter = elementConverter;
+ return this;
+ }
+
+ /** Destination table name for the DynamoDB sink. */
+ public DynamoDbSinkBuilder<InputT> setDestinationTableName(String tableName) {
+ this.tableName = tableName;
+ return this;
+ }
+
+ /**
+ * @param overwriteByPartitionKeys list of attribute key names for the sink to deduplicate on if
+ * you want to bypass the no duplication limitation of a single batch write request.
+ * Batching DynamoDB sink will drop request items in the buffer if their primary
+ * keys(composite) values are the same as the newly added ones. The newer request item in a
+ * single batch takes precedence.
+ */
+ public DynamoDbSinkBuilder<InputT> setOverwriteByPartitionKeys(
+ List<String> overwriteByPartitionKeys) {
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ return this;
+ }
+
+ public DynamoDbSinkBuilder<InputT> setFailOnError(boolean failOnError) {
+ this.failOnError = failOnError;
+ return this;
+ }
+
+ @Override
+ public DynamoDbSinkBuilder<InputT> setMaxBatchSizeInBytes(long maxBatchSizeInBytes) {
+ throw new InvalidConfigurationException(
+ "Max batch size in bytes is not supported by the DynamoDB sink implementation.");
+ }
+
+ @Override
+ public DynamoDbSinkBuilder<InputT> setMaxRecordSizeInBytes(long maxRecordSizeInBytes) {
+ throw new InvalidConfigurationException(
+ "Max record size in bytes is not supported by the DynamoDB sink implementation.");
+ }
Review Comment:
Thanks for adding this
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidRequestException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB request was invalid. */
+public class InvalidRequestException extends RuntimeException {
Review Comment:
Why is this not a sub type of `DynamoDbSinkException`?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidConfigurationException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB configuration was invalid. */
+public class InvalidConfigurationException extends RuntimeException {
Review Comment:
I see the annotations were recently removed, why was this? https://github.com/apache/flink-connector-dynamodb/pull/1/commits/598373ace67c96e1e7fb466f60129f8475738bf9
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidConfigurationException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB configuration was invalid. */
+public class InvalidConfigurationException extends RuntimeException {
Review Comment:
Why is this not a subtype of `DynamoDbSinkException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1013073396
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidRequestException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB request was invalid. */
+public class InvalidRequestException extends RuntimeException {
Review Comment:
I see the annotations were recently removed, why was this? https://github.com/apache/flink-connector-dynamodb/pull/1/commits/598373ace67c96e1e7fb466f60129f8475738bf9
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkException.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown when DynamoDb sink failed to write data. */
+public class DynamoDbSinkException extends RuntimeException {
Review Comment:
Missing `@PublicEvolving`
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidConfigurationException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB configuration was invalid. */
+public class InvalidConfigurationException extends RuntimeException {
Review Comment:
Missing `@PublicEvolving`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r999345486
##########
flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/TableRequestsContainerTest.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/** Tests for {@link TableRequestsContainer}. */
+public class TableRequestsContainerTest {
+ @Test
+ public void testRequestNotDeduplicatedWhenNoTableConfig() {
+ new Scenario()
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .witItem("table", ImmutableMap.of("pk", "1", "sk", "1"))
+ .withDescription("number of requests without deduplication configuration")
+ .withExpectedNumberOfItems(2)
+ .runScenario();
+ }
+
+ @Test
+ public void testDeduplicatedOnCompositeKey() {
+ new Scenario()
+ .withTableConfig("tableOne", "pk", "sk")
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
+ // to be deduplicated
+ .witItem("tableOne", ImmutableMap.of("pk", "1", "sk", "1"))
Review Comment:
added a test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000511892
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination table using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * sink will backpressure while the number of elements in the buffer is at the maximum
+ * <li>{@code maxBatchSizeInBytes}: the maximum size of a batch of entries that may be written to
+ * DynamoDb measured in bytes
+ * <li>{@code maxTimeInBufferMS}: the maximum amount of time an entry is allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be flushed immediately
+ * <li>{@code maxRecordSizeInBytes}: the maximum size of a record the sink will accept into the
+ * buffer, a record of size larger than this will be rejected when passed to the sink
+ * <li>{@code failOnError}: when an exception is encountered while persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * <li>{@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in the same batch request.
+ * Only the latest record with the same combination of key attributes is preserved in the
+ * request.
+ * </ul>
+ *
+ * <p>Please see the writer implementation in {@link DynamoDbSinkWriter}
+ *
+ * @param <InputT> Type of the elements handled by this sink
+ */
+@PublicEvolving
+public class DynamoDbSink<InputT> extends AsyncSinkBase<InputT, DynamoDbWriteRequest> {
+
+ private final Properties dynamoDbClientProperties;
+ private final boolean failOnError;
+ private final String tableName;
+ private List<String> overwriteByPKeys;
+
+ protected DynamoDbSink(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties) {
+ super(
+ elementConverter,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes);
+ checkNotNull(tableName, "Destination table name must be set.");
Review Comment:
How about checking for empty string too?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination table using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * sink will backpressure while the number of elements in the buffer is at the maximum
+ * <li>{@code maxBatchSizeInBytes}: the maximum size of a batch of entries that may be written to
+ * DynamoDb measured in bytes
+ * <li>{@code maxTimeInBufferMS}: the maximum amount of time an entry is allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be flushed immediately
+ * <li>{@code maxRecordSizeInBytes}: the maximum size of a record the sink will accept into the
+ * buffer, a record of size larger than this will be rejected when passed to the sink
+ * <li>{@code failOnError}: when an exception is encountered while persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * <li>{@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in the same batch request.
+ * Only the latest record with the same combination of key attributes is preserved in the
+ * request.
+ * </ul>
+ *
+ * <p>Please see the writer implementation in {@link DynamoDbSinkWriter}
+ *
+ * @param <InputT> Type of the elements handled by this sink
+ */
+@PublicEvolving
+public class DynamoDbSink<InputT> extends AsyncSinkBase<InputT, DynamoDbWriteRequest> {
+
+ private final Properties dynamoDbClientProperties;
+ private final boolean failOnError;
+ private final String tableName;
+ private List<String> overwriteByPKeys;
+
+ protected DynamoDbSink(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
Review Comment:
Should we null check this too?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(request));
+ }
+
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ unprocessed.size());
+ numRecordsOutErrorsCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ requestEntries.size(),
+ err);
+ numRecordsOutErrorsCounter.inc(requestEntries.size());
+
+ if (DynamoDbExceptionUtils.isNotRetryableException(err.getCause())) {
+ getFatalExceptionCons()
+ .accept(
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception", err));
+ } else if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ } else {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getWriteRequest().toString().getBytes(StandardCharsets.UTF_8).length;
Review Comment:
> requestEntry.getWriteRequest().toString()
What does this return? It might also be expensive to call this, depending on the implementation. I agree though this is not an easy thing to compute here, I had the same issue with the CloudWatch metrics sink
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
Review Comment:
nit: From reading this it is not clear what this does. We could rename the class to increase readability.
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(request));
+ }
+
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ unprocessed.size());
+ numRecordsOutErrorsCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ requestEntries.size(),
+ err);
+ numRecordsOutErrorsCounter.inc(requestEntries.size());
+
+ if (DynamoDbExceptionUtils.isNotRetryableException(err.getCause())) {
Review Comment:
Unfortunately the async sink base does not perform ANY error classification. This means we also need to handle other types of non-recoverable errors. We implemented some base helpers for this, please see how it is done in KDS/KDF: https://github.com/apache/flink/blob/f53dd03fd5fe87a366de0da1fdef4ad88d5b20af/flink-connectors/flink-connector-aws-kinesis-streams/src/main/java/org/apache/flink/connector/kinesis/sink/KinesisStreamsSinkWriter.java#L73
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(request));
+ }
+
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ unprocessed.size());
+ numRecordsOutErrorsCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ requestEntries.size(),
+ err);
+ numRecordsOutErrorsCounter.inc(requestEntries.size());
+
+ if (DynamoDbExceptionUtils.isNotRetryableException(err.getCause())) {
+ getFatalExceptionCons()
+ .accept(
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception", err));
+ } else if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ } else {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getWriteRequest().toString().getBytes(StandardCharsets.UTF_8).length;
+ }
Review Comment:
You need to implement `close()` and close both the DDB client and Http client
##########
flink-connector-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilderTest.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.util.Map;
+
+/** Tests for {@link DynamoDbSinkBuilder}. */
+public class DynamoDbSinkBuilderTest {
+
+ @Test
Review Comment:
We should be using junit 5 (jupiter) and Asseert4J for new code as per the [coding guide](https://flink.apache.org/contributing/code-style-and-quality-common.html#tooling). Please migrate tests
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequest.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+/**
+ * Represents a single DynamoDb {@link WriteRequest}. TODO remove this class, replace with interface
+ * to support batch and non-batch modes
+ */
+@Internal
+public class DynamoDbWriteRequest implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ private final WriteRequest writeRequest;
+
+ public DynamoDbWriteRequest(WriteRequest writeRequest) {
+ this.writeRequest = writeRequest;
+ }
+
+ public WriteRequest getWriteRequest() {
+ return writeRequest;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ DynamoDbWriteRequest that = (DynamoDbWriteRequest) o;
+ return Objects.equals(writeRequest, that.writeRequest);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(writeRequest);
+ }
Review Comment:
Why do we need these?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
Review Comment:
We *could* remove the whole retry policy thing from the initial connector and add support when we implement https://issues.apache.org/jira/browse/FLINK-29683. This way we do not need to maintain 2 sets of keys too.
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination table using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * sink will backpressure while the number of elements in the buffer is at the maximum
+ * <li>{@code maxBatchSizeInBytes}: the maximum size of a batch of entries that may be written to
+ * DynamoDb measured in bytes
+ * <li>{@code maxTimeInBufferMS}: the maximum amount of time an entry is allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be flushed immediately
+ * <li>{@code maxRecordSizeInBytes}: the maximum size of a record the sink will accept into the
+ * buffer, a record of size larger than this will be rejected when passed to the sink
+ * <li>{@code failOnError}: when an exception is encountered while persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * <li>{@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in the same batch request.
+ * Only the latest record with the same combination of key attributes is preserved in the
+ * request.
+ * </ul>
+ *
+ * <p>Please see the writer implementation in {@link DynamoDbSinkWriter}
+ *
+ * @param <InputT> Type of the elements handled by this sink
+ */
+@PublicEvolving
+public class DynamoDbSink<InputT> extends AsyncSinkBase<InputT, DynamoDbWriteRequest> {
+
+ private final Properties dynamoDbClientProperties;
+ private final boolean failOnError;
+ private final String tableName;
+ private List<String> overwriteByPKeys;
Review Comment:
This can be `final` and can we expand the `P` abbreviation to make it more readable?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/AWSDynamoDbUtil.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.connector.aws.config.AWSConfigConstants;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.runtime.util.EnvironmentInformation;
+import org.apache.flink.streaming.connectors.dynamodb.config.AWSDynamoDbConfigConstants;
+
+import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
+import software.amazon.awssdk.core.client.config.SdkAdvancedClientOption;
+import software.amazon.awssdk.core.client.config.SdkClientConfiguration;
+import software.amazon.awssdk.core.client.config.SdkClientOption;
+import software.amazon.awssdk.core.retry.RetryPolicy;
+import software.amazon.awssdk.core.retry.backoff.BackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.EqualJitterBackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.FixedDelayBackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.FullJitterBackoffStrategy;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClientBuilder;
+
+import java.net.URI;
+import java.time.Duration;
+import java.util.Optional;
+import java.util.Properties;
+
+/** Some utilities specific to Amazon DynamoDB. */
+@Internal
+public class AWSDynamoDbUtil extends AWSGeneralUtil {
Review Comment:
We do not need to extend `AWSGeneralUtil`. This was done in Kinesis for backwards compatibility
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbSerializationUtil.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Serialization Utils for DynamoDb {@link AttributeValue}. This class is currently not
+ * serializable, see <a href="https://github.com/aws/aws-sdk-java-v2/issues/3143">open issue</a>
+ */
+@Internal
+public class DynamoDbSerializationUtil {
+
+ public static void serializeWriteRequest(WriteRequest writeRequest, DataOutputStream out)
+ throws IOException {
+ if (writeRequest.putRequest() != null) {
+ out.writeUTF(DynamoDbWriteRequestType.PUT.name());
+ Map<String, AttributeValue> item = writeRequest.putRequest().item();
+ serializeItem(item, out);
+ } else if (writeRequest.deleteRequest() != null) {
+ out.writeUTF(DynamoDbWriteRequestType.DELETE.name());
+ Map<String, AttributeValue> key = writeRequest.deleteRequest().key();
+ serializeItem(key, out);
+ } else {
+ throw new IllegalArgumentException("Empty write request");
+ }
+ }
+
+ public static WriteRequest deserializeWriteRequest(DataInputStream in) throws IOException {
+ String writeRequestType = in.readUTF();
+ DynamoDbWriteRequestType dynamoDbWriteRequestType =
+ DynamoDbWriteRequestType.valueOf(writeRequestType);
+ switch (dynamoDbWriteRequestType) {
+ case PUT:
+ return WriteRequest.builder()
+ .putRequest(PutRequest.builder().item(deserializeItem(in)).build())
+ .build();
+ case DELETE:
+ return WriteRequest.builder()
+ .deleteRequest(DeleteRequest.builder().key(deserializeItem(in)).build())
+ .build();
+ default:
+ throw new IllegalArgumentException(
+ "Invalid write request type " + writeRequestType);
+ }
+ }
+
+ public static void serializeItem(Map<String, AttributeValue> item, DataOutputStream out)
+ throws IOException {
+ out.writeInt(item.size());
+ for (Map.Entry<String, AttributeValue> entry : item.entrySet()) {
+ out.writeUTF(entry.getKey());
+ AttributeValue value = entry.getValue();
+ serializeAttributeValue(value, out);
+ }
+ }
+
+ public static void serializeAttributeValue(AttributeValue value, DataOutputStream out)
+ throws IOException {
+ if (value.nul() != null) {
+ out.writeUTF(DynamoDbType.NULL.name());
Review Comment:
We can reduce state size here by using an enum ordinal or similar shortened key
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbElementConverter implements ElementConverter<String, DynamoDbWriteRequest> {
+ *
+ * @Override
+ * public DynamoDbWriteRequest apply(String s) {
+ * final Map<String, AttributeValue> item = new HashMap<>();
+ * item.put("your-key", AttributeValue.builder().s(s).build());
+ * return new DynamoDbWriteRequest(
+ * WriteRequest.builder()
+ * .putRequest(PutRequest.builder()
+ * .item(item)
+ * .build())
+ * .build()
+ * );
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setElementConverter(new DummyDynamoDbElementConverter())
+ * .setDestinationTableName("your-table-name")
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code destinationTableName} destination table for the sink
+ * <li>{@code overwriteByPKeys} will be empty meaning no records deduplication will be performed
+ * by the batch sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
+ private static final int DEFAULT_MAX_BUFFERED_REQUESTS = 10000;
+ private static final long DEFAULT_MAX_BATCH_SIZE_IN_B = 16 * 1000 * 1000;
+ private static final long DEFAULT_MAX_TIME_IN_BUFFER_MS = 5000;
+ private static final long DEFAULT_MAX_RECORD_SIZE_IN_B = 400 * 1000;
+ private static final boolean DEFAULT_FAIL_ON_ERROR = false;
+
+ private boolean failOnError;
+ private Properties dynamodbClientProperties;
+
+ private ElementConverter<InputT, DynamoDbWriteRequest> elementConverter;
+ private String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkBuilder<InputT> setDynamoDbProperties(Properties properties) {
+ this.dynamodbClientProperties = properties;
+ return this;
+ }
+
+ public DynamoDbSinkBuilder<InputT> setElementConverter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter) {
+ this.elementConverter = elementConverter;
+ return this;
+ }
+
+ /** Destination DynamoDB table name for the sink to write to. */
+ public DynamoDbSinkBuilder<InputT> setDestinationTableName(String tableName) {
+ this.tableName = tableName;
+ return this;
+ }
+
+ /**
+ * @param overwriteByPKeys provide partition key and (optionally) sort key name if you want to
+ * bypass no duplication limitation of single batch write request. Batching DynamoDB sink
+ * will drop request items in the buffer if their primary keys(composite) values are the
+ * same as newly added one.
Review Comment:
> and (optionally) sort key name
It is not clear from this comment or the method signature how I provide a sort key. If it is element 0 and 1, could we split to individual fields instead?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
Review Comment:
nit: You can use `CollectionUtil::isNullOrEmpty`
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
+ // here, as we have at most 25 elements per batch
+ return UUID.randomUUID().toString();
+ } else {
+ Map<String, AttributeValue> requestItems = getRequestItems(request);
+
+ StringBuilder builder = new StringBuilder();
+ for (String keyName : pKeys) {
+ AttributeValue value = requestItems.get(keyName);
+
+ if (value == null) {
+ throw new InvalidRequestException(
+ "Request " + request.toString() + " does not contain pKey " + keyName);
+ }
+
+ builder.append(getKeyValue(value));
+ }
+
+ return builder.toString();
+ }
+ }
+
+ /**
+ * Returns string value of a partition key attribute. Each primary key attribute must be defined
+ * as type String, Number, or binary as per DynamoDB specification.
+ */
+ private static String getKeyValue(AttributeValue value) {
+ StringBuilder builder = new StringBuilder();
+
+ if (value.n() != null) {
+ builder.append(value.n());
+ }
+
+ if (value.s() != null) {
+ builder.append(value.s());
+ }
+
+ if (value.b() != null) {
+ builder.append(value.b().asUtf8String());
+ }
+
+ return builder.toString();
Review Comment:
Do we want to validate that no unsupported types were used, or is this done elsewhere?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
+ // here, as we have at most 25 elements per batch
+ return UUID.randomUUID().toString();
+ } else {
+ Map<String, AttributeValue> requestItems = getRequestItems(request);
+
+ StringBuilder builder = new StringBuilder();
+ for (String keyName : pKeys) {
+ AttributeValue value = requestItems.get(keyName);
+
+ if (value == null) {
+ throw new InvalidRequestException(
+ "Request " + request.toString() + " does not contain pKey " + keyName);
+ }
+
+ builder.append(getKeyValue(value));
Review Comment:
We should add a separator. Consider `["ab", "cd"]` and `["a", "bcd"]` would both result in the same composite key
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbSerializationUtil.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Serialization Utils for DynamoDb {@link AttributeValue}. This class is currently not
+ * serializable, see <a href="https://github.com/aws/aws-sdk-java-v2/issues/3143">open issue</a>
+ */
+@Internal
+public class DynamoDbSerializationUtil {
+
+ public static void serializeWriteRequest(WriteRequest writeRequest, DataOutputStream out)
+ throws IOException {
+ if (writeRequest.putRequest() != null) {
+ out.writeUTF(DynamoDbWriteRequestType.PUT.name());
+ Map<String, AttributeValue> item = writeRequest.putRequest().item();
+ serializeItem(item, out);
+ } else if (writeRequest.deleteRequest() != null) {
+ out.writeUTF(DynamoDbWriteRequestType.DELETE.name());
+ Map<String, AttributeValue> key = writeRequest.deleteRequest().key();
+ serializeItem(key, out);
+ } else {
+ throw new IllegalArgumentException("Empty write request");
+ }
+ }
+
+ public static WriteRequest deserializeWriteRequest(DataInputStream in) throws IOException {
+ String writeRequestType = in.readUTF();
+ DynamoDbWriteRequestType dynamoDbWriteRequestType =
+ DynamoDbWriteRequestType.valueOf(writeRequestType);
+ switch (dynamoDbWriteRequestType) {
+ case PUT:
+ return WriteRequest.builder()
+ .putRequest(PutRequest.builder().item(deserializeItem(in)).build())
+ .build();
+ case DELETE:
+ return WriteRequest.builder()
+ .deleteRequest(DeleteRequest.builder().key(deserializeItem(in)).build())
+ .build();
+ default:
+ throw new IllegalArgumentException(
+ "Invalid write request type " + writeRequestType);
+ }
+ }
+
+ public static void serializeItem(Map<String, AttributeValue> item, DataOutputStream out)
Review Comment:
Can the rest of the methods below here be private?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
Review Comment:
There is a bunch of duplication in the `AWSDynamoDbUtil` with base classes. Consider doing this, and deleting `AWSDynamoDbUtil`:
```
AWSAsyncSinkUtil.createAwsAsyncClient(
dynamoDbClientProperties,
AWSGeneralUtil.createAsyncHttpClient(properties),
DynamoDbAsyncClient.builder(),
USER_AGENT,
USER_AGENT_KEY);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998328370
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
> I am also thinking, that if we want to really support that we probably have to remove that deduplication that we are doing in case we find a duplicate key, and instead "split" the batch, to make sure we do not lose records with the same primary key
IMO deduplication within a batch on the Flink DDB sink side is a useful feature, because it reduces writes to DDB. The current deduplication ensures the newest record takes precedence, which is the behaviour we see on consecutive `BatchWriteItem` requests.
The difference I can see here is that the ChangeDataCapture stream off the DDB table will be different depending on whether deduping happens in Flink DDB Sink. This might be possible if the user wants to write to DDB but also track number of changes in the stream. This sounds like a relatively niche use case though, since this use case is more suited to a Streaming Destination like Kinesis.
So, I think it makes sense to keep the deduping. @nirtsruya do you have a particular use case where splitting into batch would be preferable?
> But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
> If the order of the duplicate/update entries matters I think batch sink can not be used anyway, as the order is mixed up by partial write failures/full retries and parallelism in the sink. Then it is up to user to dedup before the sink.
As far as I can tell, the second request in DDB will take precedence and overwrite the first request. I can see Batching being very useful for Batch jobs where deduping already occured, and we just want to dump a large amt of data to DDB.
For streaming use case, this situation is mitigated if user sets the `MAX_IN_FLIGHT_REQUESTS` to 1, because the `AsyncSinkWriter` will put any failed items from requests to the front of the queue, so the "older" items will be retried first.
In the future, I was thinking we can implement an improvement to use `PutItem` API, and the DDB Sink can add an additional key-value with `eventTime` as the value. Whenever an item with older event time is written, we can just drop the record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998651365
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
> May be we could provide a setting to turn off deduplication on the sink side
This is the situation if `tableConfig` is not provided right? Or do you think a separate config will be required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r965965165
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbAttributeValueUtils.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbAttributeValue;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbRequest;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/** Utils class to convert this sink representation of AWS sdk2 dynamodb classes. */
+@Internal
+public class DynamoDbAttributeValueUtils {
+
+ public static WriteRequest toWriteRequest(DynamoDbRequest dynamoDbRequest) {
+ if (dynamoDbRequest.putRequest() != null) {
+ return WriteRequest.builder()
+ .putRequest(
+ PutRequest.builder()
+ .item(toAttributeValueMap(dynamoDbRequest.putRequest().item()))
+ .build())
+ .build();
+ } else if (dynamoDbRequest.deleteRequest() != null) {
Review Comment:
The `DynamoDbRequestConverter` allows creation of a `dynamoDbRequest` containing both `putRequest` and `deleteRequest`, yet the `deleteRequest` would be dropped here. Either support both or one, but needs to be considered throughout
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
It looks like we have created wrappers around the AWS SDK DynamoDB model within the sink. Can you please explain your reasoning behind this approach? The idea behind hiding the ElementConverter is to 1/ hide the internal implementation and 2/ enable sink to manage de/serialisation. However I am struggling to see the benefits here since the 2x models are very similar. If we will copy the underlying model it might be best to use that model, since users might already be creating these objects.
Did you consider using a more generic `ElementConverter`, like the [DynamoDBMapper](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.html)? I am familiar with this so not sure if there are performance/feature limitations
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbAttributeValue.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * Represents a DynamoDb Attribute value.
+ *
+ * <p>The following fields represents the different data types in dynamo db. Only <b>one</b> field
+ * should be set.
+ *
+ * <ul>
+ * <li>s - String
+ * <li>n - Number
+ * <li>b - Binary
+ * <li>bool - Boolean
+ * <li>nul - Null
+ * <li>ss - String set
+ * <li>ns - Number set
+ * <li>bs - Binary set
+ * <li>l - List of DynamoDbAttributeValue
+ * <li>m - Map of String to DynamoDbAttributeValue
+ * </ul>
+ */
+@PublicEvolving
+public class DynamoDbAttributeValue implements Serializable {
+
+ private final Boolean nul;
+ private final String s;
+ private final String n;
+ private final byte[] b;
+ private final Boolean bool;
+ private final Set<String> ss;
+ private final Set<String> ns;
+ private final Set<byte[]> bs;
+ private final List<DynamoDbAttributeValue> l;
+ private final Map<String, DynamoDbAttributeValue> m;
+
Review Comment:
How about if the user sets multiple fields? I assume this would result in some fields being dropped, but there is nothing to prevent it
##########
pom.xml:
##########
@@ -0,0 +1,1140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
Review Comment:
I will open a separate PR to include the majority of this pom separately so we can decouple from yours
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/config/DynamoDbTablesConfig.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.config;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/** Represents DynamoDB tables configuration. */
+@PublicEvolving
+public class DynamoDbTablesConfig implements Serializable {
Review Comment:
We are adding complexity here to support multiple tables. I wonder if it is best to support a single table per sink and then user would create separate operator per table. This is inline with the JDBC connector, what do you think? Do you typically use multiple tables per sink in your jobs?
##########
pom.xml:
##########
@@ -0,0 +1,1140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <parent>
+ <groupId>org.apache</groupId>
+ <artifactId>apache</artifactId>
+ <version>20</version>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connectors</artifactId>
+ <version>1.16-SNAPSHOT</version>
+ <name>Flink : Connectors : </name>
Review Comment:
?
##########
pom.xml:
##########
@@ -0,0 +1,1140 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <parent>
+ <groupId>org.apache</groupId>
+ <artifactId>apache</artifactId>
+ <version>20</version>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connectors</artifactId>
+ <version>1.16-SNAPSHOT</version>
+ <name>Flink : Connectors : </name>
+ <packaging>pom</packaging>
+ <url>https://flink.apache.org</url>
+ <inceptionYear>2022</inceptionYear>
+
+ <licenses>
+ <license>
+ <name>The Apache Software License, Version 2.0</name>
+ <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
+ <distribution>repo</distribution>
+ </license>
+ </licenses>
+
+ <scm>
+ <url>https://github.com/apache/flink-connector-dynamodb</url>
+ <connection>git@github.com:apache/flink-connector-dynamodb.git</connection>
+ <developerConnection>
+ scm:git:https://gitbox.apache.org/repos/asf/flink-connector-dynamodb.git
+ </developerConnection>
+ </scm>
+
+ <modules>
+ <module>flink-connector-aws-dynamodb</module>
+ </modules>
+
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+
+ <flink.version>1.16-SNAPSHOT</flink.version>
+ <flink.shaded.version>15.0</flink.shaded.version>
+ <flink.shaded.jackson.version>2.12.4</flink.shaded.jackson.version>
+ <target.java.version>1.8</target.java.version>
+ <maven.compiler.source>${target.java.version}</maven.compiler.source>
+ <maven.compiler.target>${target.java.version}</maven.compiler.target>
+ <scala.version>2.12.7</scala.version>
+ <scala.binary.version>2.12</scala.binary.version>
+
+ <junit4.version>4.13.2</junit4.version>
+ <junit5.version>5.8.1</junit5.version>
+ <assertj.version>3.21.0</assertj.version>
+ <archunit.version>0.22.0</archunit.version>
+ <testcontainers.version>1.16.2</testcontainers.version>
+ <mockito.version>2.21.0</mockito.version>
+ <spotless.version>2.4.2</spotless.version>
+
+ <japicmp.skip>false</japicmp.skip>
+ <japicmp.referenceVersion>1.14.0</japicmp.referenceVersion>
+ <japicmp.outputDir>tools/japicmp-output</japicmp.outputDir>
+
+ <slf4j.version>1.7.36</slf4j.version>
+ <log4j.version>2.17.2</log4j.version>
+
+ <flink.convergence.phase>validate</flink.convergence.phase>
+ <test.randomization.seed/>
+ <test.unit.pattern>**/*Test.*</test.unit.pattern>
+
+ <aws.sdk.version>2.17.249</aws.sdk.version>
+ </properties>
+
+
+
+ <dependencyManagement>
Review Comment:
Let's use the AWS BOM instead of individual dependencies, see https://github.com/apache/flink/pull/20521/files
##########
flink-connector-aws-dynamodb/pom.xml:
##########
@@ -0,0 +1,162 @@
+<?xml version="1.0" encoding="UTF-8"?>
Review Comment:
Out of the box I get compilation error when running `mvn clean package`
```
[ERROR] flink-connector-dynamodb/flink-connector-aws-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkITCase.java:[60,76] cannot find symbol
[ERROR] symbol: variable DYNAMODB
[ERROR] location: class org.apache.flink.util.DockerImageVersions
```
##########
flink-connector-aws-dynamodb/pom.xml:
##########
@@ -0,0 +1,162 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connectors</artifactId>
+ <version>1.16-SNAPSHOT</version>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>flink-connector-aws-dynamodb</artifactId>
+ <name>Flink : Connectors : Amazon DynamoDB</name>
+
+ <packaging>jar</packaging>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-shaded-force-shading</artifactId>
+ </dependency>
+ <!-- Connectors -->
Review Comment:
Fix formatting here, move comment down
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbAttributeValueUtils.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbAttributeValue;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbRequest;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/** Utils class to convert this sink representation of AWS sdk2 dynamodb classes. */
+@Internal
+public class DynamoDbAttributeValueUtils {
+
+ public static WriteRequest toWriteRequest(DynamoDbRequest dynamoDbRequest) {
Review Comment:
This method sits out of place, give the Util class is pertaining to `DynamoDbAttributeValue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#issuecomment-1240753383
Apologies for the delay, I have left some comments. As commented in the main pom I will open a PR to contribute the general repo files so we can decouple from this contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998651365
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
> May be we could provide a setting to turn off deduplication on the sink side
Currently we can do this via `tableConfig`, isn't it? Or do you think a separate config will be better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997401172
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * add elements will be blocked while the number of elements in the buffer is at the maximum
Review Comment:
fixed thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998236206
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
Thanks @sthm good comment
If I understand correcty, we are concerned that the sink uses version x of dynamodb sdk, while the user is using version y, and the versions are incompatible for some reason? I think during compile time it would be hard to detect, as this is somewhat a dependency management issue - unless we shade the aws sdk and only allow to pass these shaded objects as an input?
Otherwise, do we think an error during sink startup makes sense that the sdk classes used are incompatible with the one used in the sink due to serialization issues? similar to how flink fails if an object that is passed to the operators is not serializable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r997909454
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Hi,
Duplicated keys only matter when they get to the same batch request, then dynamodb service will fail the request. I took a simple approach to have accumulator to deduplicate entries based on PK/SK in the same request. E.g. if we had 25 entries and 1 entry has duplicated PK/SK we would then write 24 entries in the request.
But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
I may be misunderstood the question. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998595643
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/config/AWSDynamoDbConfigConstants.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.config;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+
+import software.amazon.awssdk.core.retry.backoff.EqualJitterBackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.FixedDelayBackoffStrategy;
+import software.amazon.awssdk.core.retry.backoff.FullJitterBackoffStrategy;
+
+/** Defaults for {@link AWSDynamoDbUtil}. */
+@PublicEvolving
+public class AWSDynamoDbConfigConstants {
+
+ public static final String BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT =
+ "Apache Flink %s (%s) DynamoDB Connector";
+
+ /** Identifier for user agent prefix. */
+ public static final String DYNAMODB_CLIENT_USER_AGENT_PREFIX =
+ "aws.dynamodb.client.user-agent-prefix";
+
+ public static final String AWS_DYNAMODB_CLIENT_RETRY_PREFIX = "aws.dynamodb.client.retry";
+
+ /**
+ * Possible configuration values for backoff strategy on retry when writing to DynamoDB.
+ * Internally, a corresponding implementation of {@link
+ * software.amazon.awssdk.core.retry.backoff.BackoffStrategy} will be used.
+ */
+ @PublicEvolving
+ public enum BackoffStrategy {
+
+ /**
+ * Backoff strategy that uses a full jitter strategy for computing the next backoff delay. A
+ * full jitter strategy will always compute a new random delay between and the computed
+ * exponential backoff for each subsequent request. See example: {@link
+ * FullJitterBackoffStrategy}
+ */
+ FULL_JITTER,
+
+ /**
+ * Backoff strategy that uses equal jitter for computing the delay before the next retry. An
+ * equal jitter backoff strategy will first compute an exponential delay based on the
+ * current number of retries, base delay and max delay. The final computed delay before the
+ * next retry will keep half of this computed delay plus a random delay computed as a random
+ * number between 0 and half of the exponential delay plus one. See example: {@link
+ * EqualJitterBackoffStrategy}
+ */
+ EQUAL_JITTER,
+
+ /**
+ * Simple backoff strategy that always uses a fixed delay for the delay * before the next
+ * retry attempt. See example: {@link FixedDelayBackoffStrategy}
+ */
+ FIXED_DELAY
+ }
Review Comment:
As a follow-up, we should generalise and pull this config into `aws-base`. Just a callout, nothing to do now
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
+ private static final int DEFAULT_MAX_BUFFERED_REQUESTS = 10000;
+ private static final long DEFAULT_MAX_BATCH_SIZE_IN_B = 16 * 1000 * 1000;
+ private static final long DEFAULT_MAX_TIME_IN_BUFFER_MS = 5000;
+ private static final long DEFAULT_MAX_RECORD_SIZE_IN_B = 400 * 1000;
+ private static final boolean DEFAULT_FAIL_ON_ERROR = false;
+
+ private boolean failOnError;
+ private DynamoDbTablesConfig dynamoDbTablesConfig;
+ private Properties dynamodbClientProperties;
+
+ private ElementConverter<InputT, DynamoDbWriteRequest> elementConverter;
+
+ public DynamoDbSinkBuilder<InputT> setDynamoDbProperties(Properties properties) {
+ this.dynamodbClientProperties = properties;
+ return this;
+ }
+
+ /**
+ * @param elementConverter the {@link ElementConverter} to be used for the sink
+ * @return {@link DynamoDbSinkBuilder} itself
+ */
Review Comment:
nit: Since this comment does not add anything above the method signature, consider removing, as per the [coding guidelines](https://flink.apache.org/contributing/code-style-and-quality-common.html#comments)
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream using the buffering
Review Comment:
```suggestion
* A DynamoDB Sink that performs async requests against a destination table using the buffering
```
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * add elements will be blocked while the number of elements in the buffer is at the maximum
+ * <li>{@code maxBatchSizeInBytes}: the maximum size of a batch of entries that may be written to
+ * DynamoDb measured in bytes
+ * <li>{@code maxTimeInBufferMS}: the maximum amount of time an entry is allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be flushed immediately
+ * <li>{@code maxRecordSizeInBytes}: the maximum size of a record the sink will accept into the
+ * buffer, a record of size larger than this will be rejected when passed to the sink
+ * <li>{@code failOnError}: when an exception is encountered while persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * <li>{@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in the same batch request.
+ * Only the latest record with the same combination of key attributes is preserved in the
+ * request.
+ * </ul>
+ *
+ * <p>Please see the writer implementation in {@link DynamoDbSinkWriter}
+ *
+ * @param <InputT> Type of the elements handled by this sink
+ */
+@PublicEvolving
+public class DynamoDbSink<InputT> extends AsyncSinkBase<InputT, DynamoDbWriteRequest> {
+
+ private final Properties dynamoDbClientProperties;
+ private final DynamoDbTablesConfig dynamoDbTablesConfig;
+ private final boolean failOnError;
+
+ protected DynamoDbSink(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ DynamoDbTablesConfig dynamoDbTablesConfig,
+ Properties dynamoDbClientProperties) {
+ super(
+ elementConverter,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes);
+ checkNotNull(dynamoDbTablesConfig, "Missing dynamoDbTablesConfig");
+ checkNotNull(dynamoDbClientProperties, "Missing dynamoDbClientProperties");
+ this.failOnError = failOnError;
+ this.dynamoDbTablesConfig = dynamoDbTablesConfig;
+ this.dynamoDbClientProperties = dynamoDbClientProperties;
+ }
+
+ /**
+ * Create a {@link DynamoDbSinkBuilder} to construct a new {@link DynamoDbSink}.
+ *
+ * @param <InputT> type of incoming records
+ * @return {@link DynamoDbSinkBuilder}
+ */
+ public static <InputT> DynamoDbSinkBuilder<InputT> builder() {
+ return new DynamoDbSinkBuilder<>();
+ }
+
+ @Internal
+ @Override
+ public StatefulSinkWriter<InputT, BufferedRequestState<DynamoDbWriteRequest>> createWriter(
+ InitContext context) throws IOException {
+ return new DynamoDbSinkWriter<>(
Review Comment:
nit: Can simplify with:
```
return restoreWriter(context, Collections.emptyList());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1002392190
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequest.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Represents a single Write Request to DynamoDb. Contains the item to be written as well as the
+ * type of the Write Request (PUT/DELETE)
+ */
+@PublicEvolving
+public class DynamoDbWriteRequest implements Serializable {
+
+ private final Map<String, AttributeValue> item;
+ private final DynamoDbWriteRequestType type;
+
+ private DynamoDbWriteRequest(Map<String, AttributeValue> item, DynamoDbWriteRequestType type) {
+ this.item = item;
+ this.type = type;
+ }
+
+ public Map<String, AttributeValue> getItem() {
+ return item;
+ }
+
+ public DynamoDbWriteRequestType getType() {
+ return type;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ DynamoDbWriteRequest that = (DynamoDbWriteRequest) o;
+ return item.equals(that.item) && type == that.type;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(item, type);
+ }
Review Comment:
not really needed, mainly for tests like de/serialization when comparing for equality, but I can also check specific fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1001481006
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, WriteRequest> {
Review Comment:
> I have one comment on the DynamoDbWriteRequest - it seems like a DTO, do we really need to make it an interface?
Yes I agree that an interface does not make sense here. Class is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000813463
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
+ // here, as we have at most 25 elements per batch
+ return UUID.randomUUID().toString();
+ } else {
+ Map<String, AttributeValue> requestItems = getRequestItems(request);
+
+ StringBuilder builder = new StringBuilder();
+ for (String keyName : pKeys) {
+ AttributeValue value = requestItems.get(keyName);
+
+ if (value == null) {
+ throw new InvalidRequestException(
+ "Request " + request.toString() + " does not contain pKey " + keyName);
+ }
+
+ builder.append(getKeyValue(value));
+ }
+
+ return builder.toString();
+ }
+ }
+
+ /**
+ * Returns string value of a partition key attribute. Each primary key attribute must be defined
+ * as type String, Number, or binary as per DynamoDB specification.
+ */
+ private static String getKeyValue(AttributeValue value) {
+ StringBuilder builder = new StringBuilder();
+
+ if (value.n() != null) {
+ builder.append(value.n());
+ }
+
+ if (value.s() != null) {
+ builder.append(value.s());
+ }
+
+ if (value.b() != null) {
+ builder.append(value.b().asUtf8String());
+ }
+
+ return builder.toString();
Review Comment:
DynamoDB only allows string, number or binary as key. We can do additional validation, but that would be applied on every item. Do you think it worth having this additional check given the performance hit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1001437365
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, WriteRequest> {
Review Comment:
Ok makes sense,
I have one comment on the DynamoDbWriteRequest - it seems like a DTO, do we really need to make it an interface?
would help when deserializing for example to have the same concrete class maybe for de/serialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003626614
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbElementConverter implements ElementConverter<String, DynamoDbWriteRequest> {
+ *
+ * @Override
+ * public DynamoDbWriteRequest apply(String s) {
+ * final Map<String, AttributeValue> item = new HashMap<>();
+ * item.put("your-key", AttributeValue.builder().s(s).build());
+ * return new DynamoDbWriteRequest(
+ * WriteRequest.builder()
+ * .putRequest(PutRequest.builder()
+ * .item(item)
+ * .build())
+ * .build()
+ * );
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setElementConverter(new DummyDynamoDbElementConverter())
+ * .setDestinationTableName("your-table-name")
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code destinationTableName} destination table for the sink
+ * <li>{@code overwriteByPKeys} will be empty meaning no records deduplication will be performed
+ * by the batch sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
+ private static final int DEFAULT_MAX_BUFFERED_REQUESTS = 10000;
+ private static final long DEFAULT_MAX_BATCH_SIZE_IN_B = 16 * 1000 * 1000;
+ private static final long DEFAULT_MAX_TIME_IN_BUFFER_MS = 5000;
+ private static final long DEFAULT_MAX_RECORD_SIZE_IN_B = 400 * 1000;
+ private static final boolean DEFAULT_FAIL_ON_ERROR = false;
+
+ private boolean failOnError;
+ private Properties dynamodbClientProperties;
+
+ private ElementConverter<InputT, DynamoDbWriteRequest> elementConverter;
+ private String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkBuilder<InputT> setDynamoDbProperties(Properties properties) {
+ this.dynamodbClientProperties = properties;
+ return this;
+ }
+
+ public DynamoDbSinkBuilder<InputT> setElementConverter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter) {
+ this.elementConverter = elementConverter;
+ return this;
+ }
+
+ /** Destination DynamoDB table name for the sink to write to. */
+ public DynamoDbSinkBuilder<InputT> setDestinationTableName(String tableName) {
+ this.tableName = tableName;
+ return this;
+ }
+
+ /**
+ * @param overwriteByPKeys provide partition key and (optionally) sort key name if you want to
+ * bypass no duplication limitation of single batch write request. Batching DynamoDB sink
+ * will drop request items in the buffer if their primary keys(composite) values are the
+ * same as newly added one.
Review Comment:
It's fine to use the list approach. It's more flexible, and given its similar in boto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-aws] dannycranmer commented on pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on PR #1:
URL: https://github.com/apache/flink-connector-aws/pull/1#issuecomment-1304179568
Seems like the checks are broken, raised a Jira to fix:
- https://issues.apache.org/jira/browse/FLINK-29896
LGTM, running tests locally before merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003526661
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(request));
+ }
+
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ unprocessed.size());
+ numRecordsOutErrorsCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ requestEntries.size(),
+ err);
+ numRecordsOutErrorsCounter.inc(requestEntries.size());
+
+ if (DynamoDbExceptionUtils.isNotRetryableException(err.getCause())) {
+ getFatalExceptionCons()
+ .accept(
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception", err));
+ } else if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ } else {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getWriteRequest().toString().getBytes(StandardCharsets.UTF_8).length;
Review Comment:
Hi @dannycranmer,
It is toString operation over a map: {payload=AttributeValue(S=value1), sort_key=AttributeValue(S=1), key=AttributeValue(S=2)}.
*NOTE: comment update because of implementation changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1014055821
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.aws.util.AWSAsyncSinkUtil;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.connector.base.sink.throwable.FatalExceptionClassifier;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getInvalidCredentialsExceptionClassifier;
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getSdkClientMisconfiguredExceptionClassifier;
+import static org.apache.flink.connector.base.sink.writer.AsyncSinkFatalExceptionClassifiers.getInterruptedExceptionClassifier;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ private static final FatalExceptionClassifier RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ResourceNotFoundException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to not being able to find the specified resources",
+ err));
+
+ private static final FatalExceptionClassifier CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ConditionalCheckFailedException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to failed conditional check",
+ err));
+
+ /* Validation exceptions are not retryable. See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html. */
+ private static final FatalExceptionClassifier VALIDATION_EXCEPTION_CLASSIFIER =
+ new FatalExceptionClassifier(
+ (err) ->
+ err instanceof DynamoDbException
+ && ((DynamoDbException) err)
+ .toBuilder()
+ .awsErrorDetails()
+ .errorCode()
+ .equalsIgnoreCase("ValidationException"),
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception because of DynamoDB request validation",
+ err));
+
+ private static final FatalExceptionClassifier DYNAMODB_FATAL_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.createChain(
+ getInterruptedExceptionClassifier(),
+ getInvalidCredentialsExceptionClassifier(),
+ RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER,
+ CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER,
+ VALIDATION_EXCEPTION_CLASSIFIER,
+ getSdkClientMisconfiguredExceptionClassifier());
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* A counter for the total number of records that were returned by DynamoDB as unprocessed and were retried */
+ private final Counter numRecordsReturnedAsUnprocessedCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ /* The asynchronous http client for the asynchronous DynamoDb client */
+ private final SdkAsyncHttpClient httpClient;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private final List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPartitionKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.numRecordsReturnedAsUnprocessedCounter =
+ metrics.counter("numRecordsReturnedAsUnprocessedCounter");
+ this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
+ this.client = buildClient(dynamoDbClientProperties, httpClient);
+ }
+
+ private DynamoDbAsyncClient buildClient(
+ Properties dynamoDbClientProperties, SdkAsyncHttpClient httpClient) {
+ AWSGeneralUtil.validateAwsCredentials(dynamoDbClientProperties);
+
+ return AWSAsyncSinkUtil.createAwsAsyncClient(
+ dynamoDbClientProperties,
+ httpClient,
+ DynamoDbAsyncClient.builder(),
+ DynamoDbConfigConstants.BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT,
+ DynamoDbConfigConstants.DYNAMODB_CLIENT_USER_AGENT_PREFIX);
+ }
Review Comment:
```suggestion
public DynamoDbSinkWriter(
ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
InitContext context,
int maxBatchSize,
int maxInFlightRequests,
int maxBufferedRequests,
long maxBatchSizeInBytes,
long maxTimeInBufferMS,
long maxRecordSizeInBytes,
boolean failOnError,
String tableName,
List<String> overwriteByPartitionKeys,
Properties dynamoDbClientProperties,
Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
super(
elementConverter,
context,
maxBatchSize,
maxInFlightRequests,
maxBufferedRequests,
maxBatchSizeInBytes,
maxTimeInBufferMS,
maxRecordSizeInBytes,
states);
this.failOnError = failOnError;
this.tableName = tableName;
this.overwriteByPartitionKeys = overwriteByPartitionKeys;
this.metrics = context.metricGroup();
this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
this.numRecordsReturnedAsUnprocessedCounter =
metrics.counter("numRecordsReturnedAsUnprocessedCounter");
this.httpClient = AWSGeneralUtil.createAsyncHttpClient(overrideClientProperties(dynamoDbClientProperties));
this.client = buildClient(dynamoDbClientProperties, httpClient);
}
private Properties overrideClientProperties(Properties dynamoDbClientProperties) {
Properties overridenProperties = new Properties();
overridenProperties.putAll(dynamoDbClientProperties);
// Specify HTTP1_1 protocol since DynamoDB endpoint doesn't support HTTP2
overridenProperties.put(AWSConfigConstants.HTTP_PROTOCOL_VERSION, "HTTP1_1");
return overridenProperties;
}
private DynamoDbAsyncClient buildClient(
Properties dynamoDbClientProperties, SdkAsyncHttpClient httpClient) {
AWSGeneralUtil.validateAwsCredentials(dynamoDbClientProperties);
return AWSAsyncSinkUtil.createAwsAsyncClient(
dynamoDbClientProperties,
httpClient,
DynamoDbAsyncClient.builder(),
DynamoDbConfigConstants.BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT,
DynamoDbConfigConstants.DYNAMODB_CLIENT_USER_AGENT_PREFIX);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-aws] dannycranmer merged pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer merged PR #1:
URL: https://github.com/apache/flink-connector-aws/pull/1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1013929755
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidRequestException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB request was invalid. */
+public class InvalidRequestException extends RuntimeException {
Review Comment:
fixed this now
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidConfigurationException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB configuration was invalid. */
+public class InvalidConfigurationException extends RuntimeException {
Review Comment:
please see my answer below. Fixed this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1013909334
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/InvalidRequestException.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+/** Exception is thrown if a DynamoDB request was invalid. */
+public class InvalidRequestException extends RuntimeException {
Review Comment:
I first started adding them for other exceptions that we expose now, and then I saw there are no PublicEvolving annotations for exceptions in the kinesis sink. Wasn’t sure if thats right , but decided to do similarly to existing code. I'll add them back now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1012406088
##########
flink-connector-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilderTest.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.util.Map;
+
+/** Tests for {@link DynamoDbSinkBuilder}. */
+public class DynamoDbSinkBuilderTest {
+
+ @Test
Review Comment:
This was done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1009201543
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination table using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb. DynamoDB client supports only up to 25 elements in the batch.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * sink will backpressure while the number of elements in the buffer is at the maximum
+ * <li>{@code maxBatchSizeInBytes}: the maximum size of a batch of entries that may be written to
+ * DynamoDb measured in bytes
+ * <li>{@code maxTimeInBufferMS}: the maximum amount of time an entry is allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be flushed immediately
+ * <li>{@code maxRecordSizeInBytes}: the maximum size of a record the sink will accept into the
+ * buffer, a record of size larger than this will be rejected when passed to the sink
+ * <li>{@code failOnError}: when an exception is encountered while persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * <li>{@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in the same batch request.
+ * Only the latest record with the same combination of key attributes is preserved in the
+ * request.
+ * </ul>
Review Comment:
This comment is out of sync with the recent changes to the class
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequest.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Represents a single Write Request to DynamoDb. Contains the item to be written as well as the
+ * type of the Write Request (PUT/DELETE)
+ */
+@PublicEvolving
+public class DynamoDbWriteRequest implements Serializable {
+
+ private final Map<String, AttributeValue> item;
+ private final DynamoDbWriteRequestType type;
+
+ private DynamoDbWriteRequest(Map<String, AttributeValue> item, DynamoDbWriteRequestType type) {
+ this.item = item;
+ this.type = type;
+ }
+
+ public Map<String, AttributeValue> getItem() {
+ return item;
+ }
+
+ public DynamoDbWriteRequestType getType() {
+ return type;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ DynamoDbWriteRequest that = (DynamoDbWriteRequest) o;
+ return item.equals(that.item) && type == that.type;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(item, type);
+ }
Review Comment:
If this is only required for tests, can we please delete it, [as per Flink coding guidelines](https://flink.apache.org/contributing/code-style-and-quality-java.html#equals--hashcode)?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequestType.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Represents a DynamoDb Write Request type. The following types are currently supported
+ *
+ * <ul>
+ * <li>PUT - Put Request
+ * <li>DELETE - Delete Request
+ * </ul>
+ */
+@PublicEvolving
+public enum DynamoDbWriteRequestType {
+
+ // Note: Enums have no stable hash code across different JVMs, use toByteValue() for
+ // this purpose.
+ PUT((byte) 0),
+ DELETE((byte) 1);
Review Comment:
Is there a reason to not use `ordinal()` and define your own here?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbType.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+
+/**
+ * enum representing the dynamodb types.
+ *
+ * <ul>
+ * <li>String
+ * <li>Number
+ * <li>Boolean
+ * <li>Null
+ * <li>Binary
+ * <li>String Set
+ * <li>Number Set
+ * <li>Binary Set
+ * <li>List
+ * <li>Map
+ * </ul>
+ */
+@Internal
+public enum DynamoDbType {
+
+ // Note: Enums have no stable hash code across different JVMs, use toByteValue() for
+ // this purpose.
+ STRING((byte) 0),
+ NUMBER((byte) 1),
+ BOOLEAN((byte) 2),
+ NULL((byte) 3),
+ BINARY((byte) 4),
+ STRING_SET((byte) 5),
+ NUMBER_SET((byte) 6),
+ BINARY_SET((byte) 7),
+ LIST((byte) 8),
+ MAP((byte) 9);
+
+ private final byte value;
+
+ DynamoDbType(byte value) {
+ this.value = value;
+ }
+
+ /**
+ * Returns the byte value representation of this {@link DynamoDbType}. The byte value is used
+ * for serialization and deserialization.
+ *
+ * <ul>
+ * <li>"0" represents {@link #STRING}.
+ * <li>"1" represents {@link #NUMBER}.
+ * <li>"2" represents {@link #BOOLEAN}.
+ * <li>"3" represents {@link #NULL}.
+ * <li>"4" represents {@link #BINARY}.
+ * <li>"5" represents {@link #STRING_SET}.
+ * <li>"6" represents {@link #NUMBER_SET}.
+ * <li>"7" represents {@link #BINARY_SET}.
+ * <li>"8" represents {@link #LIST}.
+ * <li>"9" represents {@link #MAP}.
+ * </ul>
+ */
+ public byte toByteValue() {
+ return value;
+ }
+
+ /**
+ * Creates a {@link DynamoDbType} from the given byte value. Each {@link DynamoDbType} has a
+ * byte value representation.
+ *
+ * @see #toByteValue() for mapping of byte value and {@link DynamoDbType}.
+ */
+ public static DynamoDbType fromByteValue(byte value) {
+ switch (value) {
+ case 0:
+ return STRING;
+ case 1:
+ return NUMBER;
+ case 2:
+ return BOOLEAN;
+ case 3:
+ return NULL;
+ case 4:
+ return BINARY;
+ case 5:
+ return STRING_SET;
+ case 6:
+ return NUMBER_SET;
+ case 7:
+ return BINARY_SET;
+ case 8:
+ return LIST;
+ case 9:
+ return MAP;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported byte value '" + value + "' for DynamoDb type.");
+ }
Review Comment:
Same question as above, why not use `ordinal()` ?
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/DynamoDbSerializationUtil.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbWriteRequest;
+import org.apache.flink.streaming.connectors.dynamodb.sink.DynamoDbWriteRequestType;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Serialization Utils for DynamoDb {@link AttributeValue}. This class is currently not
+ * serializable, see <a href="https://github.com/aws/aws-sdk-java-v2/issues/3143">open issue</a>
+ */
+@Internal
+public class DynamoDbSerializationUtil {
+
+ public static void serializeWriteRequest(
+ DynamoDbWriteRequest dynamoDbWriteRequest, DataOutputStream out) throws IOException {
+ out.writeByte(dynamoDbWriteRequest.getType().toByteValue());
+ Map<String, AttributeValue> item = dynamoDbWriteRequest.getItem();
+ serializeItem(item, out);
+ }
+
+ public static DynamoDbWriteRequest deserializeWriteRequest(DataInputStream in)
+ throws IOException {
+ int writeRequestType = in.read();
+ DynamoDbWriteRequestType dynamoDbWriteRequestType =
+ DynamoDbWriteRequestType.fromByteValue((byte) writeRequestType);
+ Map<String, AttributeValue> item = deserializeItem(in);
+ return DynamoDbWriteRequest.build().setType(dynamoDbWriteRequestType).setItem(item).build();
+ }
+
+ private static void serializeItem(Map<String, AttributeValue> item, DataOutputStream out)
+ throws IOException {
+ out.writeInt(item.size());
+ for (Map.Entry<String, AttributeValue> entry : item.entrySet()) {
+ out.writeUTF(entry.getKey());
+ AttributeValue value = entry.getValue();
+ serializeAttributeValue(value, out);
+ }
+ }
+
+ private static void serializeAttributeValue(AttributeValue value, DataOutputStream out)
+ throws IOException {
+ if (value.nul() != null) {
+ out.writeByte(DynamoDbType.NULL.toByteValue());
+ } else if (value.bool() != null) {
+ out.writeByte(DynamoDbType.BOOLEAN.toByteValue());
+ out.writeBoolean(value.bool());
+ } else if (value.s() != null) {
+ out.writeByte(DynamoDbType.STRING.toByteValue());
+ out.writeUTF(value.s());
+ } else if (value.n() != null) {
+ out.writeByte(DynamoDbType.NUMBER.toByteValue());
+ out.writeUTF(value.n());
+ } else if (value.b() != null) {
+ out.writeByte(DynamoDbType.BINARY.toByteValue());
+ out.writeInt(value.b().asByteArrayUnsafe().length);
+ out.write(value.b().asByteArrayUnsafe());
+ } else if (value.hasSs()) {
+ out.writeByte(DynamoDbType.STRING_SET.toByteValue());
+ out.writeInt(value.ss().size());
+ for (String s : value.ss()) {
+ out.writeUTF(s);
+ }
+ } else if (value.hasNs()) {
+ out.writeByte(DynamoDbType.NUMBER_SET.toByteValue());
+ out.writeInt(value.ns().size());
+ for (String s : value.ns()) {
+ out.writeUTF(s);
+ }
+ } else if (value.hasBs()) {
+ out.writeByte(DynamoDbType.BINARY_SET.toByteValue());
+ out.writeInt(value.bs().size());
+ for (SdkBytes sdkBytes : value.bs()) {
+ byte[] bytes = sdkBytes.asByteArrayUnsafe();
+ out.writeInt(bytes.length);
+ out.write(bytes);
+ }
+ } else if (value.hasL()) {
+ out.writeByte(DynamoDbType.LIST.toByteValue());
+ List<AttributeValue> l = value.l();
+ out.writeInt(l.size());
+ for (AttributeValue attributeValue : l) {
+ serializeAttributeValue(attributeValue, out);
+ }
+ } else if (value.hasM()) {
+ out.writeByte(DynamoDbType.MAP.toByteValue());
+ Map<String, AttributeValue> m = value.m();
+ serializeItem(m, out);
+ } else {
+ throw new IllegalArgumentException("Attribute value must not be empty: " + value);
+ }
+ }
+
+ private static Map<String, AttributeValue> deserializeItem(DataInputStream in)
+ throws IOException {
+ int size = in.readInt();
+ Map<String, AttributeValue> item = new HashMap<>(size);
+ for (int i = 0; i < size; i++) {
+ String key = in.readUTF();
+ AttributeValue attributeValue = deserializeAttributeValue(in);
+ item.put(key, attributeValue);
+ }
+ return item;
+ }
+
+ private static AttributeValue deserializeAttributeValue(DataInputStream in) throws IOException {
+ int type = in.read();
+ DynamoDbType dynamoDbType = DynamoDbType.fromByteValue((byte) type);
+ return deserializeAttributeValue(dynamoDbType, in);
+ }
+
+ private static AttributeValue deserializeAttributeValue(
+ DynamoDbType dynamoDbType, DataInputStream in) throws IOException {
+ switch (dynamoDbType) {
+ case NULL:
+ return AttributeValue.builder().nul(true).build();
+ case STRING:
+ return AttributeValue.builder().s(in.readUTF()).build();
+ case NUMBER:
+ return AttributeValue.builder().n(in.readUTF()).build();
+ case BOOLEAN:
+ return AttributeValue.builder().bool(in.readBoolean()).build();
+ case BINARY:
+ int length = in.readInt();
+ byte[] bytes = new byte[length];
+ in.read(bytes);
+ return AttributeValue.builder().b(SdkBytes.fromByteArray(bytes)).build();
+ case STRING_SET:
+ int stringSetSize = in.readInt();
+ Set<String> stringSet = new LinkedHashSet<>(stringSetSize);
+ for (int i = 0; i < stringSetSize; i++) {
+ stringSet.add(in.readUTF());
+ }
+ return AttributeValue.builder().ss(stringSet).build();
Review Comment:
Why use a `Set` here? Does this mean this is a non-symmetric transform? We could hit a situation where:
`!originalObject.equals(deserialize(serialize(originalObject)))`
If so, I think we should not be deduplicating here and the transform should be symmetrical
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] nirtsruya commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
nirtsruya commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1009255994
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbWriteRequest.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Represents a single Write Request to DynamoDb. Contains the item to be written as well as the
+ * type of the Write Request (PUT/DELETE)
+ */
+@PublicEvolving
+public class DynamoDbWriteRequest implements Serializable {
+
+ private final Map<String, AttributeValue> item;
+ private final DynamoDbWriteRequestType type;
+
+ private DynamoDbWriteRequest(Map<String, AttributeValue> item, DynamoDbWriteRequestType type) {
+ this.item = item;
+ this.type = type;
+ }
+
+ public Map<String, AttributeValue> getItem() {
+ return item;
+ }
+
+ public DynamoDbWriteRequestType getType() {
+ return type;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ DynamoDbWriteRequest that = (DynamoDbWriteRequest) o;
+ return item.equals(that.item) && type == that.type;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(item, type);
+ }
Review Comment:
There is this method however that is used for the sink state serializing - [AsyncSinkWriterTestUtils](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriterTestUtils.java#L46). It relies on equals I think. but if it is not required to use it I can reimplement it in our test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1012223894
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.aws.util.AWSAsyncSinkUtil;
+import org.apache.flink.connector.aws.util.AWSGeneralUtil;
+import org.apache.flink.connector.base.sink.throwable.FatalExceptionClassifier;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.PrimaryKeyBuilder;
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.ConditionalCheckFailedException;
+import software.amazon.awssdk.services.dynamodb.model.DeleteRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getInvalidCredentialsExceptionClassifier;
+import static org.apache.flink.connector.aws.util.AWSCredentialFatalExceptionClassifiers.getSdkClientMisconfiguredExceptionClassifier;
+import static org.apache.flink.connector.base.sink.writer.AsyncSinkFatalExceptionClassifiers.getInterruptedExceptionClassifier;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ private static final FatalExceptionClassifier RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ResourceNotFoundException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to not being able to find the specified resources",
+ err));
+
+ private static final FatalExceptionClassifier CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.withRootCauseOfType(
+ ConditionalCheckFailedException.class,
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception relating to failed conditional check",
+ err));
+
+ /**
+ * Validation exceptions are not retryable. See
+ * https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html.
+ */
+ private static final FatalExceptionClassifier VALIDATION_EXCEPTION_CLASSIFIER =
+ new FatalExceptionClassifier(
+ (err) ->
+ err instanceof DynamoDbException
+ && ((DynamoDbException) err)
+ .toBuilder()
+ .awsErrorDetails()
+ .errorCode()
+ .equalsIgnoreCase("ValidationException"),
+ err ->
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception because of DynamoDB request validation",
+ err));
+
+ private static final FatalExceptionClassifier DYNAMODB_FATAL_EXCEPTION_CLASSIFIER =
+ FatalExceptionClassifier.createChain(
+ getInterruptedExceptionClassifier(),
+ getInvalidCredentialsExceptionClassifier(),
+ RESOURCE_NOT_FOUND_EXCEPTION_CLASSIFIER,
+ CONDITIONAL_CHECK_FAILED_EXCEPTION_CLASSIFIER,
+ VALIDATION_EXCEPTION_CLASSIFIER,
+ getSdkClientMisconfiguredExceptionClassifier());
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsSendErrorsCounter;
+
+ /* A counter for the total number of records that were returned by DynamoDB as unprocessed*/
+ private final Counter numRecordsReturnedAsUnprocessedCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ /* The asynchronous http client for the asynchronous DynamoDb client */
+ private final SdkAsyncHttpClient httpClient;
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private final List<String> overwriteByPartitionKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPartitionKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPartitionKeys = overwriteByPartitionKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsSendErrorsCounter = metrics.getNumRecordsSendErrorsCounter();
+ this.numRecordsReturnedAsUnprocessedCounter =
+ metrics.counter("numRecordsReturnedAsUnprocessedCounter");
+ this.httpClient = AWSGeneralUtil.createAsyncHttpClient(dynamoDbClientProperties);
+ this.client =
+ AWSAsyncSinkUtil.createAwsAsyncClient(
+ dynamoDbClientProperties,
+ httpClient,
+ DynamoDbAsyncClient.builder(),
+ DynamoDbConfigConstants.BASE_DYNAMODB_USER_AGENT_PREFIX_FORMAT,
+ DynamoDbConfigConstants.DYNAMODB_CLIENT_USER_AGENT_PREFIX);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ List<WriteRequest> items = new ArrayList<>();
+
+ if (CollectionUtil.isNullOrEmpty(overwriteByPartitionKeys)) {
+ for (DynamoDbWriteRequest request : requestEntries) {
+ items.add(convertToWriteRequest(request));
+ }
+ } else {
+ // deduplication needed
+ Map<String, WriteRequest> container = new HashMap<>();
+ PrimaryKeyBuilder keyBuilder = new PrimaryKeyBuilder(overwriteByPartitionKeys);
+ for (DynamoDbWriteRequest request : requestEntries) {
+ WriteRequest req = convertToWriteRequest(request);
+ container.put(keyBuilder.build(req), req);
+ }
+ items.addAll(container.values());
+ }
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(ImmutableMap.of(tableName, items))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (!CollectionUtil.isNullOrEmpty(response.unprocessedItems())) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (String tableName : response.unprocessedItems().keySet()) {
+ for (WriteRequest writeRequest : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(convertToDynamoDbWriteRequest(writeRequest));
+ }
+ }
+
+ LOG.warn("DynamoDB Sink failed to persist and will retry {} entries.", unprocessed.size());
+ numRecordsReturnedAsUnprocessedCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist and will retry {} entries.",
+ requestEntries.size(),
+ err);
+ numRecordsSendErrorsCounter.inc(requestEntries.size());
+
+ if (isRetryable(err.getCause())) {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ private boolean isRetryable(Throwable err) {
+ if (!DYNAMODB_FATAL_EXCEPTION_CLASSIFIER.isFatal(err, getFatalExceptionCons())) {
+ return false;
+ }
+ if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ return false;
+ }
+
+ return true;
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getItem().toString().getBytes(StandardCharsets.UTF_8).length;
Review Comment:
As discussed offline, it is not practical to compute the `sizeInBytes` given that:
- We do not know how the client serializes the request payload
- This will be an expensive operation and we would potentially be serializing each record twice
We agree to:
- Return a dummy constant value here
- Follow up with an improvement to make this flush strategy and validation optional
The failure modes are already covered:
- Total batch size too large: this will be handled organically by AIMD
- Single `DynamoDbWriteRequest` too large: if we try to send a single request that is too large it will result in a fatal validation exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r966129384
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkElementConverter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbAttributeValueUtils;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * An implementation of the {@link ElementConverter} that uses the AWS DynamoDb SDK v2. The user
+ * needs to provide a {@link DynamoDbRequestConverter} of the {@code InputT} to transform it into a
+ * {@link DynamoDbWriteRequest} that may be persisted.
+ */
+@Internal
+public class DynamoDbSinkElementConverter<InputT>
Review Comment:
We had a discussion earlier on that matter [here](https://github.com/apache/flink/pull/18518#discussion_r793711778)
If we go for hiding the converter we do not leak underlying dynamodb sdk classes to the user, but we have to create a wrapper around their model. This means if the dynamodb client changes over time we will have to propagate model changes too.
I think it may be better to allow the user to define the element converter and not do any translation to write the request ourselves (like we had it before this PR: https://github.com/YuriGusev/flink/commit/de83346dc24367ff1101ae13934dfcf1fc649de2#diff-1c18ee5b3c5b4d827778ffdf4e5d2219dbcb02fc45be0d3665357633a134995c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] hlteoh37 commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
hlteoh37 commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998328370
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
> I am also thinking, that if we want to really support that we probably have to remove that deduplication that we are doing in case we find a duplicate key, and instead "split" the batch, to make sure we do not lose records with the same primary key
Hmm, I think deduplication within a batch on the Flink DDB sink side is a useful feature, because it reduces writes to DDB. The current deduplication ensures the newest record takes precedence, which is the behaviour we see on consecutive `BatchWriteItem` requests.
The difference I can see here is that the ChangeDataCapture stream off the DDB table will be different depending on whether deduping happens in Flink DDB Sink. This might be possible if the user wants to write to DDB but also track number of changes in the stream. This sounds like a relatively niche use case though, since this use case is more suited to a Streaming Destination like Kinesis.
> But if two entries with the same PK/SK get to two different in-flight batch request this is not a problem as deduplication then happens on DynamoDB side.
> If the order of the duplicate/update entries matters I think batch sink can not be used anyway, as the order is mixed up by partial write failures/full retries and parallelism in the sink. Then it is up to user to dedup before the sink.
As far as I can tell, the second request in DDB will take precedence and overwrite the first request. I can see Batching being very useful for Batch jobs where deduping already occured, and we just want to dump a large amt of data to DDB.
For streaming use case, this situation is mitigated if user sets the `MAX_IN_FLIGHT_REQUESTS` to 1, because the `AsyncSinkWriter` will put any failed items from requests to the front of the queue, so the "older" items will be retried first.
In the future, I was thinking we can implement an improvement to use `PutItem` API, and the DDB Sink can add an additional key-value with `eventTime` as the value. Whenever an item with older event time is written, we can just drop the record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998512587
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
May be we could provide a setting to turn off deduplication on the sink side. That would be on-pair with boto3 clients for dynamodb (overwrite_by_pkeys=['partition_key', 'sort_key'] config). Wdyt?
> In the future, I was thinking we can implement an improvement to use PutItem API, and the DDB Sink can add an additional key-value with eventTime as the value. Whenever an item with older event time attempts to be written, we can just drop the older record.
Yes I agree that would be a nice feature to add. We actually had it planned for [the first sink version](https://github.com/apache/flink/pull/17407/files#diff-ffcfaf3e245b27ca44831c37b908014be944626c1eac3d5acc63fa310345d937) (not based on AsyncSyncBase). There was interface that allows different types of producers (batching/non-batching) to be able to extend for this use case in the future. We can introduce something similar here and plan for adding this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r998653397
##########
flink-connector-aws-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbRequestConverter implements DynamoDbRequestConverter<String> {
+ *
+ * @Override
+ * public DynamoDbRequest apply(String s) {
+ * final Map<String, DynamoDbAttributeValue> item = new HashMap<>();
+ * item.put("your-key", DynamoDbAttributeValue.builder().s(s).build());
+ * return DynamoDbRequest.builder()
+ * .tableName("your-table-name")
+ * .putRequest(DynamoDbPutRequest.builder().item(item).build())
+ * .build();
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setDynamoDbRequestConverter(new DummyDynamoDbRequestConverter())
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code dynamoDbTablesConfig} will be empty meaning no records deduplication will be
+ * performed by the sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
Review Comment:
Yes this is right if there is no configuration it won't deduplicate anything. It's been a while I forgot I added test for this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r999352520
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSink.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBase;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.streaming.connectors.dynamodb.config.DynamoDbTablesConfig;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A DynamoDB Sink that performs async requests against a destination stream using the buffering
+ * protocol specified in {@link AsyncSinkBase}.
+ *
+ * <p>The sink internally uses a {@link
+ * software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient} to communicate with the AWS
+ * endpoint.
+ *
+ * <p>The behaviour of the buffering may be specified by providing configuration during the sink
+ * build time.
+ *
+ * <ul>
+ * <li>{@code maxBatchSize}: the maximum size of a batch of entries that may be written to
+ * DynamoDb.
+ * <li>{@code maxInFlightRequests}: the maximum number of in flight requests that may exist, if
+ * any more in flight requests need to be initiated once the maximum has been reached, then it
+ * will be blocked until some have completed
+ * <li>{@code maxBufferedRequests}: the maximum number of elements held in the buffer, requests to
+ * add elements will be blocked while the number of elements in the buffer is at the maximum
+ * <li>{@code maxBatchSizeInBytes}: the maximum size of a batch of entries that may be written to
+ * DynamoDb measured in bytes
+ * <li>{@code maxTimeInBufferMS}: the maximum amount of time an entry is allowed to live in the
+ * buffer, if any element reaches this age, the entire buffer will be flushed immediately
+ * <li>{@code maxRecordSizeInBytes}: the maximum size of a record the sink will accept into the
+ * buffer, a record of size larger than this will be rejected when passed to the sink
+ * <li>{@code failOnError}: when an exception is encountered while persisting to DynamoDb, the job
+ * will fail immediately if failOnError is set
+ * <li>{@code dynamoDbTablesConfig}: if provided for the table, the DynamoDb sink will attempt to
+ * deduplicate records with the same primary and/or secondary keys in the same batch request.
+ * Only the latest record with the same combination of key attributes is preserved in the
+ * request.
+ * </ul>
+ *
+ * <p>Please see the writer implementation in {@link DynamoDbSinkWriter}
+ *
+ * @param <InputT> Type of the elements handled by this sink
+ */
+@PublicEvolving
+public class DynamoDbSink<InputT> extends AsyncSinkBase<InputT, DynamoDbWriteRequest> {
+
+ private final Properties dynamoDbClientProperties;
+ private final DynamoDbTablesConfig dynamoDbTablesConfig;
+ private final boolean failOnError;
+
+ protected DynamoDbSink(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ DynamoDbTablesConfig dynamoDbTablesConfig,
+ Properties dynamoDbClientProperties) {
+ super(
+ elementConverter,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes);
+ checkNotNull(dynamoDbTablesConfig, "Missing dynamoDbTablesConfig");
+ checkNotNull(dynamoDbClientProperties, "Missing dynamoDbClientProperties");
+ this.failOnError = failOnError;
+ this.dynamoDbTablesConfig = dynamoDbTablesConfig;
+ this.dynamoDbClientProperties = dynamoDbClientProperties;
+ }
+
+ /**
+ * Create a {@link DynamoDbSinkBuilder} to construct a new {@link DynamoDbSink}.
+ *
+ * @param <InputT> type of incoming records
+ * @return {@link DynamoDbSinkBuilder}
+ */
+ public static <InputT> DynamoDbSinkBuilder<InputT> builder() {
+ return new DynamoDbSinkBuilder<>();
+ }
+
+ @Internal
+ @Override
+ public StatefulSinkWriter<InputT, BufferedRequestState<DynamoDbWriteRequest>> createWriter(
+ InitContext context) throws IOException {
+ return new DynamoDbSinkWriter<>(
Review Comment:
thats neat. Thanks fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000448945
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (String tableName : response.unprocessedItems().keySet()) {
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(request));
+ }
+ }
Review Comment:
good point, I missed this during refactoring. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003526661
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkWriter.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.connector.sink2.Sink.InitContext;
+import org.apache.flink.connector.base.sink.writer.AsyncSinkWriter;
+import org.apache.flink.connector.base.sink.writer.BufferedRequestState;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
+import org.apache.flink.streaming.connectors.dynamodb.util.AWSDynamoDbUtil;
+import org.apache.flink.streaming.connectors.dynamodb.util.DynamoDbExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Consumer;
+
+/**
+ * Sink writer created by {@link DynamoDbSink} to write to DynamoDB. More details on the operation
+ * of this sink writer may be found in the doc for {@link DynamoDbSink}. More details on the
+ * internals of this sink writer may be found in {@link AsyncSinkWriter}.
+ *
+ * <p>The {@link DynamoDbAsyncClient} used here may be configured in the standard way for the AWS
+ * SDK 2.x. e.g. the provision of {@code AWS_REGION}, {@code AWS_ACCESS_KEY_ID} and {@code
+ * AWS_SECRET_ACCESS_KEY} through environment variables etc.
+ */
+@Internal
+class DynamoDbSinkWriter<InputT> extends AsyncSinkWriter<InputT, DynamoDbWriteRequest> {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamoDbSinkWriter.class);
+
+ /* A counter for the total number of records that have encountered an error during put */
+ private final Counter numRecordsOutErrorsCounter;
+
+ /* The sink writer metric group */
+ private final SinkWriterMetricGroup metrics;
+
+ private final DynamoDbAsyncClient client;
+ private final boolean failOnError;
+ private final String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkWriter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter,
+ InitContext context,
+ int maxBatchSize,
+ int maxInFlightRequests,
+ int maxBufferedRequests,
+ long maxBatchSizeInBytes,
+ long maxTimeInBufferMS,
+ long maxRecordSizeInBytes,
+ boolean failOnError,
+ String tableName,
+ List<String> overwriteByPKeys,
+ Properties dynamoDbClientProperties,
+ Collection<BufferedRequestState<DynamoDbWriteRequest>> states) {
+ super(
+ elementConverter,
+ context,
+ maxBatchSize,
+ maxInFlightRequests,
+ maxBufferedRequests,
+ maxBatchSizeInBytes,
+ maxTimeInBufferMS,
+ maxRecordSizeInBytes,
+ states);
+ this.failOnError = failOnError;
+ this.tableName = tableName;
+ this.overwriteByPKeys = overwriteByPKeys;
+ this.metrics = context.metricGroup();
+ this.numRecordsOutErrorsCounter = metrics.getNumRecordsOutErrorsCounter();
+ this.client = AWSDynamoDbUtil.createClient(dynamoDbClientProperties);
+ }
+
+ @Override
+ protected void submitRequestEntries(
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResultConsumer) {
+
+ TableRequestsContainer container = new TableRequestsContainer(overwriteByPKeys);
+ requestEntries.forEach(container::put);
+
+ CompletableFuture<BatchWriteItemResponse> future =
+ client.batchWriteItem(
+ BatchWriteItemRequest.builder()
+ .requestItems(
+ ImmutableMap.of(tableName, container.getRequestItems()))
+ .build());
+
+ future.whenComplete(
+ (response, err) -> {
+ if (err != null) {
+ handleFullyFailedRequest(err, requestEntries, requestResultConsumer);
+ } else if (response.unprocessedItems() != null
+ && !response.unprocessedItems().isEmpty()) {
+ handlePartiallyUnprocessedRequest(response, requestResultConsumer);
+ } else {
+ requestResultConsumer.accept(Collections.emptyList());
+ }
+ });
+ }
+
+ private void handlePartiallyUnprocessedRequest(
+ BatchWriteItemResponse response, Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ List<DynamoDbWriteRequest> unprocessed = new ArrayList<>();
+
+ for (WriteRequest request : response.unprocessedItems().get(tableName)) {
+ unprocessed.add(new DynamoDbWriteRequest(request));
+ }
+
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ unprocessed.size());
+ numRecordsOutErrorsCounter.inc(unprocessed.size());
+
+ requestResult.accept(unprocessed);
+ }
+
+ private void handleFullyFailedRequest(
+ Throwable err,
+ List<DynamoDbWriteRequest> requestEntries,
+ Consumer<List<DynamoDbWriteRequest>> requestResult) {
+ LOG.warn(
+ "DynamoDB Sink failed to persist {} entries. Adding these entries back to retry.",
+ requestEntries.size(),
+ err);
+ numRecordsOutErrorsCounter.inc(requestEntries.size());
+
+ if (DynamoDbExceptionUtils.isNotRetryableException(err.getCause())) {
+ getFatalExceptionCons()
+ .accept(
+ new DynamoDbSinkException(
+ "Encountered non-recoverable exception", err));
+ } else if (failOnError) {
+ getFatalExceptionCons()
+ .accept(new DynamoDbSinkException.DynamoDbSinkFailFastException(err));
+ } else {
+ requestResult.accept(requestEntries);
+ }
+ }
+
+ @Override
+ protected long getSizeInBytes(DynamoDbWriteRequest requestEntry) {
+ // dynamodb calculates item size as a sum of all attributes and all values, but doing so on
+ // every operation may be too expensive, so this is just an estimate
+ return requestEntry.getWriteRequest().toString().getBytes(StandardCharsets.UTF_8).length;
Review Comment:
Hi @dannycranmer,
It returns write request serialised into string with all attribute types (pretty much like dynamodb json format). Something like
WriteRequest(PutRequest=PutRequest(Item={sk=AttributeValue(S=1), pk=AttributeValue(S=1), ...})). Actual data written is smaller, so this is over-estimation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003528260
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilder.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Properties;
+
+/**
+ * Builder to construct {@link DynamoDbSink}.
+ *
+ * <p>The following example shows the minimum setup to create a {@link DynamoDbSink} that writes
+ * records into DynamoDb
+ *
+ * <pre>{@code
+ * private static class DummyDynamoDbElementConverter implements ElementConverter<String, DynamoDbWriteRequest> {
+ *
+ * @Override
+ * public DynamoDbWriteRequest apply(String s) {
+ * final Map<String, AttributeValue> item = new HashMap<>();
+ * item.put("your-key", AttributeValue.builder().s(s).build());
+ * return new DynamoDbWriteRequest(
+ * WriteRequest.builder()
+ * .putRequest(PutRequest.builder()
+ * .item(item)
+ * .build())
+ * .build()
+ * );
+ * }
+ * }
+ * DynamoDbSink<String> dynamoDbSink = DynamoDbSink.<String>builder()
+ * .setElementConverter(new DummyDynamoDbElementConverter())
+ * .setDestinationTableName("your-table-name")
+ * .build();
+ * }</pre>
+ *
+ * <p>If the following parameters are not set in this builder, the following defaults will be used:
+ *
+ * <ul>
+ * <li>{@code maxBatchSize} will be 25
+ * <li>{@code maxInFlightRequests} will be 50
+ * <li>{@code maxBufferedRequests} will be 10000
+ * <li>{@code maxBatchSizeInBytes} will be 16 MB i.e. {@code 16 * 1000 * 1000}
+ * <li>{@code maxTimeInBufferMS} will be 5000ms
+ * <li>{@code maxRecordSizeInBytes} will be 400 KB i.e. {@code 400 * 1000}
+ * <li>{@code failOnError} will be false
+ * <li>{@code destinationTableName} destination table for the sink
+ * <li>{@code overwriteByPKeys} will be empty meaning no records deduplication will be performed
+ * by the batch sink
+ * </ul>
+ *
+ * @param <InputT> type of elements that should be persisted in the destination
+ */
+@PublicEvolving
+public class DynamoDbSinkBuilder<InputT>
+ extends AsyncSinkBaseBuilder<InputT, DynamoDbWriteRequest, DynamoDbSinkBuilder<InputT>> {
+
+ private static final int DEFAULT_MAX_BATCH_SIZE = 25;
+ private static final int DEFAULT_MAX_IN_FLIGHT_REQUESTS = 50;
+ private static final int DEFAULT_MAX_BUFFERED_REQUESTS = 10000;
+ private static final long DEFAULT_MAX_BATCH_SIZE_IN_B = 16 * 1000 * 1000;
+ private static final long DEFAULT_MAX_TIME_IN_BUFFER_MS = 5000;
+ private static final long DEFAULT_MAX_RECORD_SIZE_IN_B = 400 * 1000;
+ private static final boolean DEFAULT_FAIL_ON_ERROR = false;
+
+ private boolean failOnError;
+ private Properties dynamodbClientProperties;
+
+ private ElementConverter<InputT, DynamoDbWriteRequest> elementConverter;
+ private String tableName;
+
+ private List<String> overwriteByPKeys;
+
+ public DynamoDbSinkBuilder<InputT> setDynamoDbProperties(Properties properties) {
+ this.dynamodbClientProperties = properties;
+ return this;
+ }
+
+ public DynamoDbSinkBuilder<InputT> setElementConverter(
+ ElementConverter<InputT, DynamoDbWriteRequest> elementConverter) {
+ this.elementConverter = elementConverter;
+ return this;
+ }
+
+ /** Destination DynamoDB table name for the sink to write to. */
+ public DynamoDbSinkBuilder<InputT> setDestinationTableName(String tableName) {
+ this.tableName = tableName;
+ return this;
+ }
+
+ /**
+ * @param overwriteByPKeys provide partition key and (optionally) sort key name if you want to
+ * bypass no duplication limitation of single batch write request. Batching DynamoDB sink
+ * will drop request items in the buffer if their primary keys(composite) values are the
+ * same as newly added one.
Review Comment:
@dannycranmer do you think it is ok to keep this way or you would like to change the approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] dannycranmer commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
dannycranmer commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1003629002
##########
flink-connector-dynamodb/src/main/java/org/apache/flink/streaming/connectors/dynamodb/util/PrimaryKeyBuilder.java:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.streaming.connectors.dynamodb.sink.InvalidRequestException;
+import org.apache.flink.util.CollectionUtil;
+
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+/** Helper to construct primary(composite) key for a DynamoDB request. */
+@Internal
+public class PrimaryKeyBuilder {
+
+ public static String build(List<String> pKeys, WriteRequest request) {
+
+ if (CollectionUtil.isNullOrEmpty(pKeys)) {
+ // fake key, because no dynamodb partition key or sort key provided. Using UUID should be safe
+ // here, as we have at most 25 elements per batch
+ return UUID.randomUUID().toString();
+ } else {
+ Map<String, AttributeValue> requestItems = getRequestItems(request);
+
+ StringBuilder builder = new StringBuilder();
+ for (String keyName : pKeys) {
+ AttributeValue value = requestItems.get(keyName);
+
+ if (value == null) {
+ throw new InvalidRequestException(
+ "Request " + request.toString() + " does not contain pKey " + keyName);
+ }
+
+ builder.append(getKeyValue(value));
+ }
+
+ return builder.toString();
+ }
+ }
+
+ /**
+ * Returns string value of a partition key attribute. Each primary key attribute must be defined
+ * as type String, Number, or binary as per DynamoDB specification.
+ */
+ private static String getKeyValue(AttributeValue value) {
+ StringBuilder builder = new StringBuilder();
+
+ if (value.n() != null) {
+ builder.append(value.n());
+ }
+
+ if (value.s() != null) {
+ builder.append(value.s());
+ }
+
+ if (value.b() != null) {
+ builder.append(value.b().asUtf8String());
+ }
+
+ return builder.toString();
Review Comment:
We already validate that the `AttributeValue` is not null above. How about we validate the return of this method is not empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-connector-dynamodb] YuriGusev commented on a diff in pull request #1: [FLINK-24229][Connectors][DynamoDB] - Add AWS DynamoDB connector
Posted by GitBox <gi...@apache.org>.
YuriGusev commented on code in PR #1:
URL: https://github.com/apache/flink-connector-dynamodb/pull/1#discussion_r1000815682
##########
flink-connector-dynamodb/src/test/java/org/apache/flink/streaming/connectors/dynamodb/sink/DynamoDbSinkBuilderTest.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.dynamodb.sink;
+
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Test;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.util.Map;
+
+/** Tests for {@link DynamoDbSinkBuilder}. */
+public class DynamoDbSinkBuilderTest {
+
+ @Test
Review Comment:
Didn't see that changed. Thanks, I'll migrate it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org