You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/08/17 00:47:46 UTC
[GitHub] [hudi] zhedoubushishi opened a new pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
zhedoubushishi opened a new pull request #3486:
URL: https://github.com/apache/hudi/pull/3486
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d63997a11e4201f0b2879b7d42c3ee4310a5c1fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117)
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 6e5fcdb7578f89e181caa4775626fa1951891e41 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-964406414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231)
* a62e9caa37881ea1e439b4d046bca5c05616b438 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-971649163
@zhedoubushishi : awesome, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-956563212
> @zhedoubushishi : Do you think you will find time to address feedback. We are looking to see if we can get this in for the upcoming release. would appreciate if you can spare some time addressing the feedback.
Sorry was working on other things. I'll try to make an update today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-962684612
Another thing I found is the Azure test failed, but it passed in my local environment, not sure why...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 946772d9878df27bdd3dac8ece5dc09b8fc4b681 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779)
* 3e74eb4c2ae0e22677b6c972a97be516fd352eab Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 946772d9878df27bdd3dac8ece5dc09b8fc4b681 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-971782526
@zhedoubushishi : thanks for the contribution! good job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r741605659
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
I see your point. changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740846337
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -164,6 +171,52 @@
.withDocumentation("Key name under base_path at which to create a ZNode and acquire lock. "
+ "Final path on zk will look like base_path/lock_key. We recommend setting this to the table name");
+ public static final ConfigProperty<String> DYNAMODB_TABLE_NAME = ConfigProperty
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-961589641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-965093176
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* a62e9caa37881ea1e439b4d046bca5c05616b438 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253)
* 64833f78880f515071e3723d3a37dcb921b9ef14 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-967315371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
}, {
"hash" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 64833f78880f515071e3723d3a37dcb921b9ef14 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280)
* b68fe583d825dcda1710e6deea75e6f3882852d9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-965254737
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 64833f78880f515071e3723d3a37dcb921b9ef14 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 946772d9878df27bdd3dac8ece5dc09b8fc4b681 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-963678812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* e26d0e0fee083d88ae5948c616df803ec88059b3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192)
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r743769523
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -41,6 +41,11 @@
<artifactId>hudi-timeline-service</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review comment:
wondering if this is needed here, at all?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -35,6 +38,13 @@
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_CONNECTION_TIMEOUT_MS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_SESSION_TIMEOUT_MS;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_LOCK_BILLING_MODE_PROP_KEY;
Review comment:
We have the ability to group the configs on docs together i.e `HoodieLockConfig` + `DynamoDBLockConfig` so if this is the reasons for the dependency, we can actually move this back to `hoodie-aws`?
##########
File path: packaging/hudi-flink-bundle/pom.xml
##########
@@ -149,6 +150,10 @@
<include>org.apache.hbase:hbase-protocol</include>
<include>org.apache.htrace:htrace-core</include>
<include>commons-codec:commons-codec</include>
+
Review comment:
I think just adding these into the bundles are sufficient, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-963678812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* e26d0e0fee083d88ae5948c616df803ec88059b3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192)
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740786182
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieConfigAWSCredentialsProvider.java
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.BasicSessionCredentials;
+import org.apache.hudi.config.HoodieAWSConfig;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Properties;
+
+public class HoodieConfigAWSCredentialsProvider implements AWSCredentialsProvider {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieConfigAWSCredentialsProvider.class);
+
+ private AWSCredentials awsCredentials;
+
+ public HoodieConfigAWSCredentialsProvider(Properties props) {
+ String accessKey = props.getProperty(HoodieAWSConfig.AWS_ACCESS_KEY.key(), null);
Review comment:
Good point. Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r741536160
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -218,6 +222,27 @@
<version>${zk-curator.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
Review comment:
@vinothchandar : I assume you are suggesting to create a new module hudi-aws and move these over there and so all bundles (spark, utilities, flink etc) can depend on them and leave client-common as is.
@zhedoubushishi : can you address above feedback please.
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -41,6 +41,11 @@
<artifactId>hudi-timeline-service</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review comment:
here is what is making this necessary.
We add HoodieAwsconfig in HoodieWriteconfig and hence client-common has to depend on hudi-aws module. If we can follow something like kafka-connect-configs, we can keep HoodieAwsConfig separate and not add it to HoodieWriteConfig, we should be able to get rid of this dep.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -35,6 +38,13 @@
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_CONNECTION_TIMEOUT_MS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_SESSION_TIMEOUT_MS;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_LOCK_BILLING_MODE_PROP_KEY;
Review comment:
if we go with the approach I suggested above, these configs and constants will move to hudi-aws module and we can make hudi-aws depend on hudi-client-common(for the TBL_NAME usage above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-964439598
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* a62e9caa37881ea1e439b4d046bca5c05616b438 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-965143987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* a62e9caa37881ea1e439b4d046bca5c05616b438 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253)
* 64833f78880f515071e3723d3a37dcb921b9ef14 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 093275425688b2572da5e857899fecbc0c718cf2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057)
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910a7114c24bdb872eab1f5132921bc0e8a2d9aa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785)
* ae59be1af39c07bfdd3874df0cc18d3671255e31 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-938662308
@zhedoubushishi : I see "WIP" in title of the patch? May I know if its ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-971651387
@zhedoubushishi : I took precedence from existing locking classes. We have LockConfiguration and HoodieLockConfig. should be fine I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 093275425688b2572da5e857899fecbc0c718cf2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057)
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r741535577
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
what I meant is, we should catch LockNotGrantedException and return false. so that caller might retry again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r743769523
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -41,6 +41,11 @@
<artifactId>hudi-timeline-service</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review comment:
wondering if this is needed here, at all?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -35,6 +38,13 @@
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_CONNECTION_TIMEOUT_MS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_SESSION_TIMEOUT_MS;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_LOCK_BILLING_MODE_PROP_KEY;
Review comment:
We have the ability to group the configs on docs together i.e `HoodieLockConfig` + `DynamoDBLockConfig` so if this is the reasons for the dependency, we can actually move this back to `hoodie-aws`?
##########
File path: packaging/hudi-flink-bundle/pom.xml
##########
@@ -149,6 +150,10 @@
<include>org.apache.hbase:hbase-protocol</include>
<include>org.apache.htrace:htrace-core</include>
<include>commons-codec:commons-codec</include>
+
Review comment:
I think just adding these into the bundles are sufficient, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-961589641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-965267826
@zhedoubushishi : hey wenning. I spent sometime with the patch and have removed hudi-client-common -> hudi-aws dependency. But introduced the dep other way (hudi-aws -> hudi-client-common). I ensured atleast compilation succeeds and looks like CI succeeds too. Can you take a look at my latest commit. also, if you can test it once on your end, would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-965093176
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* a62e9caa37881ea1e439b4d046bca5c05616b438 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253)
* 64833f78880f515071e3723d3a37dcb921b9ef14 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-967354438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
}, {
"hash" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3340",
"triggerID" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* b68fe583d825dcda1710e6deea75e6f3882852d9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae59be1af39c07bfdd3874df0cc18d3671255e31 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788)
* d63997a11e4201f0b2879b7d42c3ee4310a5c1fa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 6e5fcdb7578f89e181caa4775626fa1951891e41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118)
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 4444742a4edf98827acd52d27a8b3d4a8ec0b27b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d63997a11e4201f0b2879b7d42c3ee4310a5c1fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117)
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 6e5fcdb7578f89e181caa4775626fa1951891e41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118)
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae59be1af39c07bfdd3874df0cc18d3671255e31 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 946772d9878df27bdd3dac8ece5dc09b8fc4b681 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779)
* 3e74eb4c2ae0e22677b6c972a97be516fd352eab Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784)
* 910a7114c24bdb872eab1f5132921bc0e8a2d9aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-967324862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
}, {
"hash" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3340",
"triggerID" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 64833f78880f515071e3723d3a37dcb921b9ef14 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280)
* b68fe583d825dcda1710e6deea75e6f3882852d9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r741604415
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -218,6 +222,27 @@
<version>${zk-curator.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
Review comment:
Done. Created ```hudi-aws``` module
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
I see your point. changed.
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -218,6 +222,27 @@
<version>${zk-curator.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
Review comment:
Done. Created ```hudi-aws``` module
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
I see your point. changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r741604415
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -218,6 +222,27 @@
<version>${zk-curator.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
Review comment:
Done. Created ```hudi-aws``` module
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
I see your point. changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r744317945
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -35,6 +38,13 @@
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_CONNECTION_TIMEOUT_MS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_SESSION_TIMEOUT_MS;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_LOCK_BILLING_MODE_PROP_KEY;
Review comment:
This part of code is depend on ```HoodieWriteConfig```:
```
public static final ConfigProperty<String> DYNAMODB_LOCK_PARTITION_KEY = ConfigProperty
.key(DYNAMODB_BASED_LOCK_PROPERTY_PREFIX + "partition_key")
.noDefaultValue()
.withInferFunction(cfg -> {
if (cfg.contains(HoodieWriteConfig.TBL_NAME)) {
return Option.of(cfg.getString(HoodieWriteConfig.TBL_NAME));
}
return Option.empty();
})
.withDocumentation("For DynamoDB based lock provider, the partition key for the DynamoDB lock table. "
+ "Each Hudi dataset should has it's unique key so concurrent writers could refer to the same partition key."
+ " By default we use the Hudi table name specified to be the partition key");
```
So I didn't move it to ```hoodie-aws```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-962691777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
* e26d0e0fee083d88ae5948c616df803ec88059b3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-965254737
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 64833f78880f515071e3723d3a37dcb921b9ef14 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 4444742a4edf98827acd52d27a8b3d4a8ec0b27b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 946772d9878df27bdd3dac8ece5dc09b8fc4b681 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910a7114c24bdb872eab1f5132921bc0e8a2d9aa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785)
* ae59be1af39c07bfdd3874df0cc18d3671255e31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-956557401
@zhedoubushishi : Do you think you will find time to address feedback. We are looking to see if we can get this in for the upcoming release. would appreciate if you can spare some time addressing the feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r741604415
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -218,6 +222,27 @@
<version>${zk-curator.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
Review comment:
Done. Created ```hudi-aws``` module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r741535577
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
what I meant is, we should catch LockNotGrantedException and return false. so that caller might retry again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-964406414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231)
* a62e9caa37881ea1e439b4d046bca5c05616b438 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-972221709
> @zhedoubushishi : thanks for the contribution! good job.
Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-956563212
> @zhedoubushishi : Do you think you will find time to address feedback. We are looking to see if we can get this in for the upcoming release. would appreciate if you can spare some time addressing the feedback.
Sorry was working on other things. I'll try to make an update today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740797832
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
Review comment:
Yes L175 is only called in tests, removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740755755
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -275,6 +300,45 @@
<skip>false</skip>
</configuration>
</plugin>
+ <plugin>
+ <groupId>io.fabric8</groupId>
+ <artifactId>docker-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>prepare-it-database</id>
+ <phase>pre-integration-test</phase>
+ <goals>
+ <goal>start</goal>
+ </goals>
+ <configuration>
+ <images>
+ <image>
+ <name>amazon/dynamodb-local:${dynamodb-local.version}</name>
Review comment:
It's because ```ITTestDynamoDBBasedLockProvider``` is located under ```hudi-client```, ```TestZookeeperBasedLockProvider``` is also located under ```hudi-client```. Do you want me to relocate it to ```hudi-integ-test```?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieConfigAWSCredentialsProvider.java
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.BasicSessionCredentials;
+import org.apache.hudi.config.HoodieAWSConfig;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Properties;
+
+public class HoodieConfigAWSCredentialsProvider implements AWSCredentialsProvider {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieConfigAWSCredentialsProvider.class);
+
+ private AWSCredentials awsCredentials;
+
+ public HoodieConfigAWSCredentialsProvider(Properties props) {
+ String accessKey = props.getProperty(HoodieAWSConfig.AWS_ACCESS_KEY.key(), null);
Review comment:
Good point. Changed.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieAWSCredentialsProviderFactory.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSCredentialsProviderChain;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+
+public class HoodieAWSCredentialsProviderFactory {
+ public static AWSCredentialsProvider getAwsCredentialsProvider(Properties props) {
+ return getAwsCredentialsProviderChain(props);
+ }
+
+ private static AWSCredentialsProvider getAwsCredentialsProviderChain(Properties props) {
+ List<AWSCredentialsProvider> providers = new ArrayList<>();
+ providers.add(new HoodieConfigAWSCredentialsProvider(props));
+ providers.add(new DefaultAWSCredentialsProviderChain());
Review comment:
Yes your understanding is correct.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
+ createLockTableInDynamoDB(dynamoDB, tableName, "PAY_PER_REQUEST", 0L, 0L);
+ }
+
+ private static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName, String billingMode,
+ Long readCapacityUnits, Long writeCapacityUnits) {
+ KeySchemaElement partitionKeyElement = new KeySchemaElement();
+ partitionKeyElement.setAttributeName("key");
+ partitionKeyElement.setKeyType(KeyType.HASH);
+
+ List<KeySchemaElement> keySchema = new ArrayList<>();
+ keySchema.add(partitionKeyElement);
+
+ Collection<AttributeDefinition> attributeDefinitions = new ArrayList<>();
+ attributeDefinitions.add(new AttributeDefinition().withAttributeName("key").withAttributeType(ScalarAttributeType.S));
+
+ CreateTableRequest createTableRequest = new CreateTableRequest(tableName, keySchema);
+ createTableRequest.setAttributeDefinitions(attributeDefinitions);
+ createTableRequest.setBillingMode(billingMode);
+ if (billingMode.equals("PROVISIONED")) {
Review comment:
Done
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
Review comment:
Yes L175 is only called in tests, removed.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -164,6 +171,52 @@
.withDocumentation("Key name under base_path at which to create a ZNode and acquire lock. "
+ "Final path on zk will look like base_path/lock_key. We recommend setting this to the table name");
+ public static final ConfigProperty<String> DYNAMODB_TABLE_NAME = ConfigProperty
Review comment:
Done
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
> If lock is not granted, why do we throw exception here? LockManager has a retry mechanism through which it will try to acquire lock few times with some delays inbetween. Don't think we can throw from here if lock could not be acquired.
If we don't catch ```LockNotGrantedException```, then ```LockManager``` also won't catch this exception: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L75, then the whole program would crush w/o retry logic.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
If we don't catch ```LockNotGrantedException```, then ```LockManager``` also won't catch this exception: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L75, then the whole program would crush w/o retry logic.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
If we don't catch ```LockNotGrantedException```, then ```LockManager``` also won't catch this exception: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L75, then the whole program would crush w/o retry logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740755755
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -275,6 +300,45 @@
<skip>false</skip>
</configuration>
</plugin>
+ <plugin>
+ <groupId>io.fabric8</groupId>
+ <artifactId>docker-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>prepare-it-database</id>
+ <phase>pre-integration-test</phase>
+ <goals>
+ <goal>start</goal>
+ </goals>
+ <configuration>
+ <images>
+ <image>
+ <name>amazon/dynamodb-local:${dynamodb-local.version}</name>
Review comment:
It's because ```ITTestDynamoDBBasedLockProvider``` is located under ```hudi-client```, ```TestZookeeperBasedLockProvider``` is also located under ```hudi-client```. Do you want me to relocate it to ```hudi-integ-test```?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieConfigAWSCredentialsProvider.java
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.BasicSessionCredentials;
+import org.apache.hudi.config.HoodieAWSConfig;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Properties;
+
+public class HoodieConfigAWSCredentialsProvider implements AWSCredentialsProvider {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieConfigAWSCredentialsProvider.class);
+
+ private AWSCredentials awsCredentials;
+
+ public HoodieConfigAWSCredentialsProvider(Properties props) {
+ String accessKey = props.getProperty(HoodieAWSConfig.AWS_ACCESS_KEY.key(), null);
Review comment:
Good point. Changed.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieAWSCredentialsProviderFactory.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSCredentialsProviderChain;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+
+public class HoodieAWSCredentialsProviderFactory {
+ public static AWSCredentialsProvider getAwsCredentialsProvider(Properties props) {
+ return getAwsCredentialsProviderChain(props);
+ }
+
+ private static AWSCredentialsProvider getAwsCredentialsProviderChain(Properties props) {
+ List<AWSCredentialsProvider> providers = new ArrayList<>();
+ providers.add(new HoodieConfigAWSCredentialsProvider(props));
+ providers.add(new DefaultAWSCredentialsProviderChain());
Review comment:
Yes your understanding is correct.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
+ createLockTableInDynamoDB(dynamoDB, tableName, "PAY_PER_REQUEST", 0L, 0L);
+ }
+
+ private static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName, String billingMode,
+ Long readCapacityUnits, Long writeCapacityUnits) {
+ KeySchemaElement partitionKeyElement = new KeySchemaElement();
+ partitionKeyElement.setAttributeName("key");
+ partitionKeyElement.setKeyType(KeyType.HASH);
+
+ List<KeySchemaElement> keySchema = new ArrayList<>();
+ keySchema.add(partitionKeyElement);
+
+ Collection<AttributeDefinition> attributeDefinitions = new ArrayList<>();
+ attributeDefinitions.add(new AttributeDefinition().withAttributeName("key").withAttributeType(ScalarAttributeType.S));
+
+ CreateTableRequest createTableRequest = new CreateTableRequest(tableName, keySchema);
+ createTableRequest.setAttributeDefinitions(attributeDefinitions);
+ createTableRequest.setBillingMode(billingMode);
+ if (billingMode.equals("PROVISIONED")) {
Review comment:
Done
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
Review comment:
Yes L175 is only called in tests, removed.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -164,6 +171,52 @@
.withDocumentation("Key name under base_path at which to create a ZNode and acquire lock. "
+ "Final path on zk will look like base_path/lock_key. We recommend setting this to the table name");
+ public static final ConfigProperty<String> DYNAMODB_TABLE_NAME = ConfigProperty
Review comment:
Done
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
> If lock is not granted, why do we throw exception here? LockManager has a retry mechanism through which it will try to acquire lock few times with some delays inbetween. Don't think we can throw from here if lock could not be acquired.
If we don't catch ```LockNotGrantedException```, then ```LockManager``` also won't catch this exception: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L75, then the whole program would crush w/o retry logic.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
If we don't catch ```LockNotGrantedException```, then ```LockManager``` also won't catch this exception: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L75, then the whole program would crush w/o retry logic.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
If we don't catch ```LockNotGrantedException```, then ```LockManager``` also won't catch this exception: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L75, then the whole program would crush w/o retry logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-962683651
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
* e26d0e0fee083d88ae5948c616df803ec88059b3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-967314653
> @zhedoubushishi : hey wenning. I spent sometime with the patch and have removed hudi-client-common -> hudi-aws dependency. But introduced the dep other way (hudi-aws -> hudi-client-common). I ensured atleast compilation succeeds and looks like CI succeeds too. Can you take a look at my latest commit. also, if you can test it once on your end, would be good.
Thanks a lot! I pushed another commit to clean the code, I wonder shall we move ```AWSLockConfiguration``` to ```HoodieAWSConfig```?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e74eb4c2ae0e22677b6c972a97be516fd352eab Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784)
* 910a7114c24bdb872eab1f5132921bc0e8a2d9aa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r743769523
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -41,6 +41,11 @@
<artifactId>hudi-timeline-service</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
Review comment:
wondering if this is needed here, at all?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -35,6 +38,13 @@
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_CONNECTION_TIMEOUT_MS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_SESSION_TIMEOUT_MS;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_LOCK_BILLING_MODE_PROP_KEY;
Review comment:
We have the ability to group the configs on docs together i.e `HoodieLockConfig` + `DynamoDBLockConfig` so if this is the reasons for the dependency, we can actually move this back to `hoodie-aws`?
##########
File path: packaging/hudi-flink-bundle/pom.xml
##########
@@ -149,6 +150,10 @@
<include>org.apache.hbase:hbase-protocol</include>
<include>org.apache.htrace:htrace-core</include>
<include>commons-codec:commons-codec</include>
+
Review comment:
I think just adding these into the bundles are sufficient, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r725711054
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -275,6 +300,45 @@
<skip>false</skip>
</configuration>
</plugin>
+ <plugin>
+ <groupId>io.fabric8</groupId>
+ <artifactId>docker-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>prepare-it-database</id>
+ <phase>pre-integration-test</phase>
+ <goals>
+ <goal>start</goal>
+ </goals>
+ <configuration>
+ <images>
+ <image>
+ <name>amazon/dynamodb-local:${dynamodb-local.version}</name>
Review comment:
may I know whats the purpose of this. Is it to run dynamoDb based tests with integration tests?
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -275,6 +300,45 @@
<skip>false</skip>
</configuration>
</plugin>
+ <plugin>
+ <groupId>io.fabric8</groupId>
+ <artifactId>docker-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>prepare-it-database</id>
+ <phase>pre-integration-test</phase>
+ <goals>
+ <goal>start</goal>
+ </goals>
+ <configuration>
+ <images>
+ <image>
+ <name>amazon/dynamodb-local:${dynamodb-local.version}</name>
Review comment:
is it not possible to restrict these just to integ test module if my above statement is true.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -164,6 +171,52 @@
.withDocumentation("Key name under base_path at which to create a ZNode and acquire lock. "
+ "Final path on zk will look like base_path/lock_key. We recommend setting this to the table name");
+ public static final ConfigProperty<String> DYNAMODB_TABLE_NAME = ConfigProperty
Review comment:
lets prefix all config properties with "DYNAMODB_LOCK_*"
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
+ createLockTableInDynamoDB(dynamoDB, tableName, "PAY_PER_REQUEST", 0L, 0L);
+ }
+
+ private static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName, String billingMode,
+ Long readCapacityUnits, Long writeCapacityUnits) {
+ KeySchemaElement partitionKeyElement = new KeySchemaElement();
+ partitionKeyElement.setAttributeName("key");
+ partitionKeyElement.setKeyType(KeyType.HASH);
+
+ List<KeySchemaElement> keySchema = new ArrayList<>();
+ keySchema.add(partitionKeyElement);
+
+ Collection<AttributeDefinition> attributeDefinitions = new ArrayList<>();
+ attributeDefinitions.add(new AttributeDefinition().withAttributeName("key").withAttributeType(ScalarAttributeType.S));
+
+ CreateTableRequest createTableRequest = new CreateTableRequest(tableName, keySchema);
+ createTableRequest.setAttributeDefinitions(attributeDefinitions);
+ createTableRequest.setBillingMode(billingMode);
+ if (billingMode.equals("PROVISIONED")) {
Review comment:
there isn't any enum for billing mode in aws sdk is it?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
+ createLockTableInDynamoDB(dynamoDB, tableName, "PAY_PER_REQUEST", 0L, 0L);
+ }
+
+ private static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName, String billingMode,
+ Long readCapacityUnits, Long writeCapacityUnits) {
+ KeySchemaElement partitionKeyElement = new KeySchemaElement();
+ partitionKeyElement.setAttributeName("key");
+ partitionKeyElement.setKeyType(KeyType.HASH);
+
+ List<KeySchemaElement> keySchema = new ArrayList<>();
+ keySchema.add(partitionKeyElement);
+
+ Collection<AttributeDefinition> attributeDefinitions = new ArrayList<>();
+ attributeDefinitions.add(new AttributeDefinition().withAttributeName("key").withAttributeType(ScalarAttributeType.S));
+
+ CreateTableRequest createTableRequest = new CreateTableRequest(tableName, keySchema);
+ createTableRequest.setAttributeDefinitions(attributeDefinitions);
+ createTableRequest.setBillingMode(billingMode);
+ if (billingMode.equals("PROVISIONED")) {
+ createTableRequest.setProvisionedThroughput(
+ new ProvisionedThroughput().withReadCapacityUnits(readCapacityUnits).withWriteCapacityUnits(writeCapacityUnits));
+ }
+ dynamoDB.createTable(createTableRequest);
+
+ LOG.info("Creating dynamoDB table " + tableName + ", waiting for table to be active");
+ try {
+ TableUtils.waitUntilActive(dynamoDB, tableName);
Review comment:
can't we configure a max wait time here?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
Review comment:
lets fetch the value of lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY) and store it in class variable and reuse it.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
+ createLockTableInDynamoDB(dynamoDB, tableName, "PAY_PER_REQUEST", 0L, 0L);
+ }
+
+ private static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName, String billingMode,
+ Long readCapacityUnits, Long writeCapacityUnits) {
+ KeySchemaElement partitionKeyElement = new KeySchemaElement();
+ partitionKeyElement.setAttributeName("key");
+ partitionKeyElement.setKeyType(KeyType.HASH);
+
+ List<KeySchemaElement> keySchema = new ArrayList<>();
+ keySchema.add(partitionKeyElement);
+
+ Collection<AttributeDefinition> attributeDefinitions = new ArrayList<>();
+ attributeDefinitions.add(new AttributeDefinition().withAttributeName("key").withAttributeType(ScalarAttributeType.S));
Review comment:
can we declare a constant for "key" and reuse it.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieAWSCredentialsProviderFactory.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSCredentialsProviderChain;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+
+public class HoodieAWSCredentialsProviderFactory {
Review comment:
java docs
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieConfigAWSCredentialsProvider.java
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.auth.BasicSessionCredentials;
+import org.apache.hudi.config.HoodieAWSConfig;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Properties;
+
+public class HoodieConfigAWSCredentialsProvider implements AWSCredentialsProvider {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieConfigAWSCredentialsProvider.class);
+
+ private AWSCredentials awsCredentials;
+
+ public HoodieConfigAWSCredentialsProvider(Properties props) {
+ String accessKey = props.getProperty(HoodieAWSConfig.AWS_ACCESS_KEY.key(), null);
Review comment:
you don't need to set null as default value explicitly. props.getProperty(HoodieAWSConfig.AWS_ACCESS_KEY.key()) will return null if not found.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
+ createLockTableInDynamoDB(dynamoDB, tableName, "PAY_PER_REQUEST", 0L, 0L);
+ }
+
+ private static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName, String billingMode,
+ Long readCapacityUnits, Long writeCapacityUnits) {
+ KeySchemaElement partitionKeyElement = new KeySchemaElement();
+ partitionKeyElement.setAttributeName("key");
+ partitionKeyElement.setKeyType(KeyType.HASH);
+
+ List<KeySchemaElement> keySchema = new ArrayList<>();
+ keySchema.add(partitionKeyElement);
+
+ Collection<AttributeDefinition> attributeDefinitions = new ArrayList<>();
+ attributeDefinitions.add(new AttributeDefinition().withAttributeName("key").withAttributeType(ScalarAttributeType.S));
+
+ CreateTableRequest createTableRequest = new CreateTableRequest(tableName, keySchema);
+ createTableRequest.setAttributeDefinitions(attributeDefinitions);
+ createTableRequest.setBillingMode(billingMode);
+ if (billingMode.equals("PROVISIONED")) {
+ createTableRequest.setProvisionedThroughput(
+ new ProvisionedThroughput().withReadCapacityUnits(readCapacityUnits).withWriteCapacityUnits(writeCapacityUnits));
+ }
+ dynamoDB.createTable(createTableRequest);
+
+ LOG.info("Creating dynamoDB table " + tableName + ", waiting for table to be active");
+ try {
+ TableUtils.waitUntilActive(dynamoDB, tableName);
+ } catch (TableUtils.TableNeverTransitionedToStateException e) {
+ throw new HoodieLockException("Created dynamoDB table never transits to active", e);
+ } catch (InterruptedException e) {
+ throw new HoodieLockException("Thread interrupted while waiting for dynamoDB table to turn active", e);
+ }
+ LOG.info("Created dynamoDB table " + tableName);
+ }
+
+ private void checkRequiredProps(final LockConfiguration config) {
+ ValidationUtils.checkArgument(config.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY) != null);
+ ValidationUtils.checkArgument(config.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY) != null);
+ ValidationUtils.checkArgument(config.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY) != null);
+ ValidationUtils.checkArgument(config.getConfig().getString(DYNAMODB_REGION_PROP_KEY) != null);
+ }
+
+ private String generateLogSuffixString() {
+ String dynamoDbPartitionKey = this.lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY);
Review comment:
can we fetch the partition key in constructor and store in a local instance variable rather than fetching from config everytime
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
Review comment:
lets not create a constructor used only in tests. Can we do something like this. Split the first constructor into two. Once you construct the AmazonDynamoDBLockClient, you can call into another constructor. That way, source code path also flows thru the 2nd constructor.
Do you think this would work ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
Review comment:
minor. getDynamoDbClient
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
Review comment:
can we move static methods to a util class.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieAWSCredentialsProviderFactory.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSCredentialsProviderChain;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+
+public class HoodieAWSCredentialsProviderFactory {
+ public static AWSCredentialsProvider getAwsCredentialsProvider(Properties props) {
+ return getAwsCredentialsProviderChain(props);
+ }
+
+ private static AWSCredentialsProvider getAwsCredentialsProviderChain(Properties props) {
+ List<AWSCredentialsProvider> providers = new ArrayList<>();
+ providers.add(new HoodieConfigAWSCredentialsProvider(props));
+ providers.add(new DefaultAWSCredentialsProviderChain());
Review comment:
Can you throw some light here please.
adding two providers mean, we first try with HoodieConfigAWSCredentialsProvider and if its not available, we will fallback to using DefaultAWSCredentialsProvider is it ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
Review comment:
curious as to why local table creation is not called in this constructor
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
If lock is not granted, why do we throw exception here? LockManager has a retry mechanism through which it will try to acquire lock few times with some delays inbetween. Don't think we can throw from here if lock could not be acquired.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
Review comment:
I see this method L175 is called only in tests. Should we move this to test module only.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
+ createLockTableInDynamoDB(dynamoDB, tableName, "PAY_PER_REQUEST", 0L, 0L);
+ }
+
+ private static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName, String billingMode,
+ Long readCapacityUnits, Long writeCapacityUnits) {
+ KeySchemaElement partitionKeyElement = new KeySchemaElement();
+ partitionKeyElement.setAttributeName("key");
+ partitionKeyElement.setKeyType(KeyType.HASH);
+
+ List<KeySchemaElement> keySchema = new ArrayList<>();
+ keySchema.add(partitionKeyElement);
+
+ Collection<AttributeDefinition> attributeDefinitions = new ArrayList<>();
+ attributeDefinitions.add(new AttributeDefinition().withAttributeName("key").withAttributeType(ScalarAttributeType.S));
+
+ CreateTableRequest createTableRequest = new CreateTableRequest(tableName, keySchema);
+ createTableRequest.setAttributeDefinitions(attributeDefinitions);
+ createTableRequest.setBillingMode(billingMode);
+ if (billingMode.equals("PROVISIONED")) {
Review comment:
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/model/BillingMode.html
can we use the enum rather than hardcoding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740794691
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
+ }
+ return lock != null && !lock.isExpired();
+ }
+
+ @Override
+ public void unlock() {
+ try {
+ LOG.info(generateLogStatement(LockState.RELEASING, generateLogSuffixString()));
+ if (lock == null) {
+ return;
+ }
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ LOG.info(generateLogStatement(LockState.RELEASED, generateLogSuffixString()));
+ } catch (Exception e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()), e);
+ }
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (lock != null) {
+ if (!client.releaseLock(lock)) {
+ LOG.warn("The lock has already been stolen");
+ }
+ lock = null;
+ }
+ this.client.close();
+ } catch (Exception e) {
+ LOG.error(generateLogStatement(LockState.FAILED_TO_RELEASE, generateLogSuffixString()));
+ }
+ }
+
+ @Override
+ public LockItem getLock() {
+ return lock;
+ }
+
+ private AmazonDynamoDB getDynamoClient() {
+ String region = this.lockConfiguration.getConfig().getString(DYNAMODB_REGION_PROP_KEY);
+ String endpointURL = RegionUtils.getRegion(region).getServiceEndpoint(AmazonDynamoDB.ENDPOINT_PREFIX);
+ AwsClientBuilder.EndpointConfiguration dynamodbEndpoint =
+ new AwsClientBuilder.EndpointConfiguration(endpointURL, region);
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(dynamodbEndpoint)
+ .withCredentials(HoodieAWSCredentialsProviderFactory.getAwsCredentialsProvider(lockConfiguration.getConfig()))
+ .build();
+ }
+
+ public static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName) {
+ createLockTableInDynamoDB(dynamoDB, tableName, "PAY_PER_REQUEST", 0L, 0L);
+ }
+
+ private static void createLockTableInDynamoDB(AmazonDynamoDB dynamoDB, String tableName, String billingMode,
+ Long readCapacityUnits, Long writeCapacityUnits) {
+ KeySchemaElement partitionKeyElement = new KeySchemaElement();
+ partitionKeyElement.setAttributeName("key");
+ partitionKeyElement.setKeyType(KeyType.HASH);
+
+ List<KeySchemaElement> keySchema = new ArrayList<>();
+ keySchema.add(partitionKeyElement);
+
+ Collection<AttributeDefinition> attributeDefinitions = new ArrayList<>();
+ attributeDefinitions.add(new AttributeDefinition().withAttributeName("key").withAttributeType(ScalarAttributeType.S));
+
+ CreateTableRequest createTableRequest = new CreateTableRequest(tableName, keySchema);
+ createTableRequest.setAttributeDefinitions(attributeDefinitions);
+ createTableRequest.setBillingMode(billingMode);
+ if (billingMode.equals("PROVISIONED")) {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740851556
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
> If lock is not granted, why do we throw exception here? LockManager has a retry mechanism through which it will try to acquire lock few times with some delays inbetween. Don't think we can throw from here if lock could not be acquired.
If we don't catch ```LockNotGrantedException```, then ```LockManager``` also won't catch this exception: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L75, then the whole program would crush w/o retry logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 4444742a4edf98827acd52d27a8b3d4a8ec0b27b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382)
* 093275425688b2572da5e857899fecbc0c718cf2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740755755
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -275,6 +300,45 @@
<skip>false</skip>
</configuration>
</plugin>
+ <plugin>
+ <groupId>io.fabric8</groupId>
+ <artifactId>docker-maven-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>prepare-it-database</id>
+ <phase>pre-integration-test</phase>
+ <goals>
+ <goal>start</goal>
+ </goals>
+ <configuration>
+ <images>
+ <image>
+ <name>amazon/dynamodb-local:${dynamodb-local.version}</name>
Review comment:
It's because ```ITTestDynamoDBBasedLockProvider``` is located under ```hudi-client```, ```TestZookeeperBasedLockProvider``` is also located under ```hudi-client```. Do you want me to relocate it to ```hudi-integ-test```?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-962691777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
* e26d0e0fee083d88ae5948c616df803ec88059b3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-962694352
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* e26d0e0fee083d88ae5948c616df803ec88059b3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-963714760
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-964355864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231)
* a62e9caa37881ea1e439b4d046bca5c05616b438 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-971121196
@nsivabalan I tested the DynamoDB based lock w/ EMR environment, insert two commits simultaneously, both commits succeeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d63997a11e4201f0b2879b7d42c3ee4310a5c1fa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117)
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 6e5fcdb7578f89e181caa4775626fa1951891e41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r730112564
##########
File path: hudi-client/hudi-client-common/pom.xml
##########
@@ -218,6 +222,27 @@
<version>${zk-curator.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
Review comment:
instead of having this dependency in `hudi-client-common`, can we have this in a `hudi-aws` module, which gets bundled with spark and utilities bundles?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -164,6 +171,52 @@
.withDocumentation("Key name under base_path at which to create a ZNode and acquire lock. "
+ "Final path on zk will look like base_path/lock_key. We recommend setting this to the table name");
+ public static final ConfigProperty<String> DYNAMODB_TABLE_NAME = ConfigProperty
Review comment:
Configs could remain here or move out as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e74eb4c2ae0e22677b6c972a97be516fd352eab Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784)
* 910a7114c24bdb872eab1f5132921bc0e8a2d9aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-963674272
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* e26d0e0fee083d88ae5948c616df803ec88059b3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192)
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r744317782
##########
File path: packaging/hudi-flink-bundle/pom.xml
##########
@@ -149,6 +150,10 @@
<include>org.apache.hbase:hbase-protocol</include>
<include>org.apache.htrace:htrace-core</include>
<include>commons-codec:commons-codec</include>
+
Review comment:
> @umehrot2 just one comment on avoiding the `hoodie-aws` dependency in `client-common`. otherwise LGTM given you have already tested this out e2e. let me know what you and @zhedoubushishi think
```HoodieWriteConfig``` is located in ```client-common```, I added ```HoodeAWSConfig``` there:
```
writeConfig.setDefaultOnCondition(!isAWSConfigSet,
HoodieAWSConfig.newBuilder().fromProperties(writeConfig.getProps()).build());
```
That's the reason why ```client-common``` depends on ```hudi-aws```.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java
##########
@@ -35,6 +38,13 @@
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_CONNECTION_TIMEOUT_MS;
import static org.apache.hudi.common.config.LockConfiguration.DEFAULT_ZK_SESSION_TIMEOUT_MS;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_LOCK_BILLING_MODE_PROP_KEY;
Review comment:
This part of code depends on ```HoodieWriteConfig```:
```
public static final ConfigProperty<String> DYNAMODB_LOCK_PARTITION_KEY = ConfigProperty
.key(DYNAMODB_BASED_LOCK_PROPERTY_PREFIX + "partition_key")
.noDefaultValue()
.withInferFunction(cfg -> {
if (cfg.contains(HoodieWriteConfig.TBL_NAME)) {
return Option.of(cfg.getString(HoodieWriteConfig.TBL_NAME));
}
return Option.empty();
})
.withDocumentation("For DynamoDB based lock provider, the partition key for the DynamoDB lock table. "
+ "Each Hudi dataset should has it's unique key so concurrent writers could refer to the same partition key."
+ " By default we use the Hudi table name specified to be the partition key");
```
So I didn't move it to ```hoodie-aws```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-964486584
The Azure test finally passed, please take a final review when you have time @vinothchandar @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-967324862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
}, {
"hash" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3340",
"triggerID" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 64833f78880f515071e3723d3a37dcb921b9ef14 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280)
* b68fe583d825dcda1710e6deea75e6f3882852d9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3340)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 093275425688b2572da5e857899fecbc0c718cf2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-956557401
@zhedoubushishi : Do you think you will find time to address feedback. We are looking to see if we can get this in for the upcoming release. would appreciate if you can spare some time addressing the feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 093275425688b2572da5e857899fecbc0c718cf2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057)
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-961589641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-961589641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 6e5fcdb7578f89e181caa4775626fa1951891e41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910a7114c24bdb872eab1f5132921bc0e8a2d9aa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 093275425688b2572da5e857899fecbc0c718cf2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057)
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-956557401
@zhedoubushishi : Do you think you will find time to address feedback. We are looking to see if we can get this in for the upcoming release. would appreciate if you can spare some time addressing the feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-962683651
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 248651db6b258d501cc0c5752a9b888137bff669 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093)
* e26d0e0fee083d88ae5948c616df803ec88059b3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r744317782
##########
File path: packaging/hudi-flink-bundle/pom.xml
##########
@@ -149,6 +150,10 @@
<include>org.apache.hbase:hbase-protocol</include>
<include>org.apache.htrace:htrace-core</include>
<include>commons-codec:commons-codec</include>
+
Review comment:
> @umehrot2 just one comment on avoiding the `hoodie-aws` dependency in `client-common`. otherwise LGTM given you have already tested this out e2e. let me know what you and @zhedoubushishi think
```HoodieWriteConfig``` is located in ```client-common```, I added ```HoodeAWSConfig``` there:
```
writeConfig.setDefaultOnCondition(!isAWSConfigSet,
HoodieAWSConfig.newBuilder().fromProperties(writeConfig.getProps()).build());
```
That's the reason why ```client-common``` is depend on ```hudi-aws```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-963674272
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* e26d0e0fee083d88ae5948c616df803ec88059b3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192)
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-962694352
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* e26d0e0fee083d88ae5948c616df803ec88059b3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-964355864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231)
* a62e9caa37881ea1e439b4d046bca5c05616b438 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-963714760
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* fce252eb9aadfc865b38da4f2cb2d8588b05f42a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-964439598
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* a62e9caa37881ea1e439b4d046bca5c05616b438 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-965143987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* a62e9caa37881ea1e439b4d046bca5c05616b438 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253)
* 64833f78880f515071e3723d3a37dcb921b9ef14 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-967315371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3057",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
}, {
"hash" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "210aa90b7cedc691b11d7e146a94ab199874ae50",
"triggerType" : "PUSH"
}, {
"hash" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0876ffb9762eda7a914e0ac8978284726cc0b267",
"triggerType" : "PUSH"
}, {
"hash" : "248651db6b258d501cc0c5752a9b888137bff669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3093",
"triggerID" : "248651db6b258d501cc0c5752a9b888137bff669",
"triggerType" : "PUSH"
}, {
"hash" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3192",
"triggerID" : "e26d0e0fee083d88ae5948c616df803ec88059b3",
"triggerType" : "PUSH"
}, {
"hash" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3231",
"triggerID" : "fce252eb9aadfc865b38da4f2cb2d8588b05f42a",
"triggerType" : "PUSH"
}, {
"hash" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3253",
"triggerID" : "a62e9caa37881ea1e439b4d046bca5c05616b438",
"triggerType" : "PUSH"
}, {
"hash" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280",
"triggerID" : "64833f78880f515071e3723d3a37dcb921b9ef14",
"triggerType" : "PUSH"
}, {
"hash" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b68fe583d825dcda1710e6deea75e6f3882852d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 210aa90b7cedc691b11d7e146a94ab199874ae50 UNKNOWN
* 0876ffb9762eda7a914e0ac8978284726cc0b267 UNKNOWN
* 64833f78880f515071e3723d3a37dcb921b9ef14 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3280)
* b68fe583d825dcda1710e6deea75e6f3882852d9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-956563212
> @zhedoubushishi : Do you think you will find time to address feedback. We are looking to see if we can get this in for the upcoming release. would appreciate if you can spare some time addressing the feedback.
Sorry was working on other things. I'll try to make an update today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
}, {
"hash" : "093275425688b2572da5e857899fecbc0c718cf2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "093275425688b2572da5e857899fecbc0c718cf2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 4444742a4edf98827acd52d27a8b3d4a8ec0b27b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2382)
* 093275425688b2572da5e857899fecbc0c718cf2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740791800
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/aws/HoodieAWSCredentialsProviderFactory.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.aws;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSCredentialsProviderChain;
+import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+
+public class HoodieAWSCredentialsProviderFactory {
+ public static AWSCredentialsProvider getAwsCredentialsProvider(Properties props) {
+ return getAwsCredentialsProviderChain(props);
+ }
+
+ private static AWSCredentialsProvider getAwsCredentialsProviderChain(Properties props) {
+ List<AWSCredentialsProvider> providers = new ArrayList<>();
+ providers.add(new HoodieConfigAWSCredentialsProvider(props));
+ providers.add(new DefaultAWSCredentialsProviderChain());
Review comment:
Yes your understanding is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
zhedoubushishi commented on a change in pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#discussion_r740851789
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.RegionUtils;
+import com.amazonaws.services.dynamodbv2.AcquireLockOptions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClientOptions;
+import com.amazonaws.services.dynamodbv2.LockItem;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.LockNotGrantedException;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.util.TableUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.aws.HoodieAWSCredentialsProviderFactory;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieLockException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_BILLING_MODE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_PARTITION_KEY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_READ_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_REGION_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_TABLE_NAME_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.DYNAMODB_WRITE_CAPACITY_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+/**
+ * A DynamoDB based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DynamoDB. Users need to have access to AWS DynamoDB to be able to use this lock.
+ */
+@NotThreadSafe
+public class DynamoDBBasedLockProvider implements LockProvider<LockItem> {
+
+ private static final Logger LOG = LogManager.getLogger(DynamoDBBasedLockProvider.class);
+
+ private final AmazonDynamoDBLockClient client;
+ private final long leaseDuration;
+ private final String tableName;
+ protected LockConfiguration lockConfiguration;
+ private volatile LockItem lock;
+
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration conf) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ AmazonDynamoDB dynamoDB = getDynamoClient();
+ // build the dynamoDb lock client
+ this.client = new AmazonDynamoDBLockClient(
+ AmazonDynamoDBLockClientOptions.builder(dynamoDB, tableName)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .withLeaseDuration(leaseDuration)
+ .withHeartbeatPeriod(leaseDuration / 3)
+ .withCreateHeartbeatBackgroundThread(true)
+ .build());
+ if (!client.lockTableExists()) {
+ createLockTableInDynamoDB(
+ dynamoDB,
+ tableName,
+ lockConfiguration.getConfig().getString(DYNAMODB_BILLING_MODE_PROP_KEY),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_READ_CAPACITY_PROP_KEY)),
+ Long.parseLong(lockConfiguration.getConfig().getString(DYNAMODB_WRITE_CAPACITY_PROP_KEY)));
+ }
+ }
+
+ // Only used for testing
+ public DynamoDBBasedLockProvider(final LockConfiguration lockConfiguration, final AmazonDynamoDBLockClient client) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ this.tableName = lockConfiguration.getConfig().getString(DYNAMODB_TABLE_NAME_PROP_KEY);
+ this.leaseDuration = Long.parseLong(lockConfiguration.getConfig().getString(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY));
+ this.client = client;
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ LOG.info(generateLogStatement(LockState.ACQUIRING, generateLogSuffixString()));
+ long millisTime = unit.toMillis(time);
+ try {
+ lock = client.acquireLock(AcquireLockOptions.builder(lockConfiguration.getConfig().getString(DYNAMODB_PARTITION_KEY_PROP_KEY))
+ .withAdditionalTimeToWaitForLock(millisTime - leaseDuration > 0 ? millisTime - leaseDuration : 0)
+ .withTimeUnit(TimeUnit.MILLISECONDS)
+ .build());
+ LOG.info(generateLogStatement(LockState.ACQUIRED, generateLogSuffixString()));
+ } catch (InterruptedException | LockNotGrantedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE, generateLogSuffixString()), e);
Review comment:
If we don't catch ```LockNotGrantedException```, then ```LockManager``` also won't catch this exception: https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L75, then the whole program would crush w/o retry logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 946772d9878df27bdd3dac8ece5dc09b8fc4b681 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779)
* 3e74eb4c2ae0e22677b6c972a97be516fd352eab UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3486: [HUDI-2314] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3486:
URL: https://github.com/apache/hudi/pull/3486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314][WIP] Add support for DynamoDb based lock
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3486:
URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1779",
"triggerID" : "946772d9878df27bdd3dac8ece5dc09b8fc4b681",
"triggerType" : "PUSH"
}, {
"hash" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1784",
"triggerID" : "3e74eb4c2ae0e22677b6c972a97be516fd352eab",
"triggerType" : "PUSH"
}, {
"hash" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1785",
"triggerID" : "910a7114c24bdb872eab1f5132921bc0e8a2d9aa",
"triggerType" : "PUSH"
}, {
"hash" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1788",
"triggerID" : "ae59be1af39c07bfdd3874df0cc18d3671255e31",
"triggerType" : "PUSH"
}, {
"hash" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2117",
"triggerID" : "d63997a11e4201f0b2879b7d42c3ee4310a5c1fa",
"triggerType" : "PUSH"
}, {
"hash" : "d2b00796c9564088aa8533431c73251993f688d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b00796c9564088aa8533431c73251993f688d4",
"triggerType" : "PUSH"
}, {
"hash" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118",
"triggerID" : "6e5fcdb7578f89e181caa4775626fa1951891e41",
"triggerType" : "PUSH"
}, {
"hash" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "99853468aec1becd1112c0ffba6ccf5f604e713d",
"triggerType" : "PUSH"
}, {
"hash" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4444742a4edf98827acd52d27a8b3d4a8ec0b27b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN
* 6e5fcdb7578f89e181caa4775626fa1951891e41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2118)
* 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN
* 4444742a4edf98827acd52d27a8b3d4a8ec0b27b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org