You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/09 17:26:17 UTC
[GitHub] [hudi] fengjian428 opened a new pull request, #6071: [Hudi 4065] create FileBasedLockProvider
fengjian428 opened a new pull request, #6071:
URL: https://github.com/apache/hudi/pull/6071
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1179582378
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b12f205de8b1d4483d05cd54db43f71352ccb493 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186203197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef589521e7635f9eb2d015b4c5e9b19d95710528 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957)
* 5fa8809f397422bbed089749b6ee75de0f9ad9d4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r921418357
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static final Logger LOG = LogManager.getLogger(TestFileBasedLockProvider.class);
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @Test
+ public void testAcquireLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ }
+
+ @Test
+ public void testReentrantLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ try {
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ Assertions.fail();
+ } catch (HoodieLockException e) {
+ // expected
+ }
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnlockWithoutLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ fileBasedLockProvider.unlock();
+ }
Review Comment:
refactored TestHoodieClientMultiWriter to test FileSystemBasedLockProvider in it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1179632928
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b12f205de8b1d4483d05cd54db43f71352ccb493 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a diff in pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r919349241
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static final Logger LOG = LogManager.getLogger(TestFileBasedLockProvider.class);
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @Test
+ public void testAcquireLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ }
+
+ @Test
+ public void testReentrantLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ try {
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ Assertions.fail();
+ } catch (HoodieLockException e) {
+ // expected
+ }
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnlockWithoutLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ fileBasedLockProvider.unlock();
+ }
Review Comment:
Just test if can unlock without lock and no exception throws
btw, I found MiniDFSCluster cannot be used in Hudi-common-client due to some conflict in pom. can I move UT to Hudi-spark-client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r922643570
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null)
+ + Path.SEPARATOR + HoodieTableMetaClient.METAFOLDER_NAME;
+ }
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + Path.SEPARATOR + LOCK_FILE_NAME);
+ this.fs = FSUtils.getFs(this.lockFile.toString(), configuration);
+ }
+
+ @Override
+ public void close() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ fs.delete(this.lockFile, true);
+ } catch (IOException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE), e);
+ }
+ }
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ try {
+ synchronized (LOCK_FILE_NAME) {
+ if (fs.exists(this.lockFile)) {
+ // Check whether lock is already expired or not, if so try to delete lock file
+ if (lockTimeoutMinutes != 0 && checkIfExpired()) {
Review Comment:
move `lockTimeoutMinutes != 0` into `checkIfExpired ` looks better?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileSystemBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieLockException;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @AfterAll
+ public static void cleanUpAfterAll() throws IOException {
+ Path workDir = dfsCluster.getFileSystem().getWorkingDirectory();
+ FileSystem fs = workDir.getFileSystem(hdfsTestService.getHadoopConf());
+ fs.delete(new Path("/tmp/lock"), true);
+ fs.delete(new Path("/tmp/.hoodie/lock"), true);
+ if (hdfsTestService != null) {
+ hdfsTestService.stop();
+ hdfsTestService = null;
+ }
+ }
+
+ @AfterEach
+ public void cleanUpAfterEach() throws IOException {
+ Path workDir = dfsCluster.getFileSystem().getWorkingDirectory();
+ FileSystem fs = workDir.getFileSystem(hdfsTestService.getHadoopConf());
+ fs.delete(new Path("/tmp/lock"), true);
+ }
+
+ @Test
+ public void testAcquireLock() {
+ FileSystemBasedLockProvider fileBasedLockProvider = new FileSystemBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testAcquireLockWithDefaultPath() {
+ lockConfiguration.getConfig().remove(FILESYSTEM_LOCK_PATH_PROP_KEY);
+ lockConfiguration.getConfig().setProperty(HoodieWriteConfig.BASE_PATH.key(), "/tmp/");
+ FileSystemBasedLockProvider fileBasedLockProvider = new FileSystemBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ lockConfiguration.getConfig().setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ }
+
+ @Test
+ public void testUnLock() {
+ FileSystemBasedLockProvider fileBasedLockProvider = new FileSystemBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
Review Comment:
assert if the result of `tryLock` is true?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileSystemBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieLockException;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @AfterAll
+ public static void cleanUpAfterAll() throws IOException {
+ Path workDir = dfsCluster.getFileSystem().getWorkingDirectory();
+ FileSystem fs = workDir.getFileSystem(hdfsTestService.getHadoopConf());
+ fs.delete(new Path("/tmp/lock"), true);
+ fs.delete(new Path("/tmp/.hoodie/lock"), true);
Review Comment:
Can we just delete the `/tmp` dir recursively?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186218982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa8809f397422bbed089749b6ee75de0f9ad9d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r920843519
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -108,6 +109,12 @@ public class HoodieLockConfig extends HoodieConfig {
.sinceVersion("0.8.0")
Review Comment:
Yes, we need to get the table's base path so that we can call `HoodieTableMetadata#getMetadataTableBasePath`.
Let me think about it.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -108,6 +109,12 @@ public class HoodieLockConfig extends HoodieConfig {
.sinceVersion("0.8.0")
Review Comment:
Yes, we need to get the table's base path so that we can call `HoodieTableMetadata.getMetadataTableBasePath`.
Let me think about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r923298238
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null)
Review Comment:
You have checked the two config items. There should be must one non-null. So IMO, here we do not need to give a default `null` value right?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null)
+ + Path.SEPARATOR + HoodieTableMetaClient.METAFOLDER_NAME;
+ }
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + Path.SEPARATOR + LOCK_FILE_NAME);
+ this.fs = FSUtils.getFs(this.lockFile.toString(), configuration);
+ }
+
+ @Override
+ public void close() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ fs.delete(this.lockFile, true);
+ } catch (IOException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE), e);
+ }
+ }
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ try {
+ synchronized (LOCK_FILE_NAME) {
+ if (fs.exists(this.lockFile)) {
+ // Check whether lock is already expired or not, if so try to delete lock file
+ if (checkIfExpired()) {
Review Comment:
Can we merge them into one condition? e.g. `a && b`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -106,7 +107,13 @@ public class HoodieLockConfig extends HoodieConfig {
.key(FILESYSTEM_LOCK_PATH_PROP_KEY)
.noDefaultValue()
.sinceVersion("0.8.0")
- .withDocumentation("For DFS based lock providers, path to store the locks under.");
+ .withDocumentation("For DFS based lock providers, path to store the locks under. If don't provide one, will use Table's meta path as default");
Review Comment:
Here we call `.noDefaultValue()`, so can we remove `If don't provide one, `, just say: `use Table's meta path as default.`?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null)
+ + Path.SEPARATOR + HoodieTableMetaClient.METAFOLDER_NAME;
+ }
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + Path.SEPARATOR + LOCK_FILE_NAME);
+ this.fs = FSUtils.getFs(this.lockFile.toString(), configuration);
+ }
+
+ @Override
+ public void close() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ fs.delete(this.lockFile, true);
+ } catch (IOException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE), e);
+ }
+ }
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ try {
+ synchronized (LOCK_FILE_NAME) {
+ if (fs.exists(this.lockFile)) {
+ // Check whether lock is already expired or not, if so try to delete lock file
Review Comment:
Remove `or not` looks more reasonable?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null)
+ + Path.SEPARATOR + HoodieTableMetaClient.METAFOLDER_NAME;
+ }
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + Path.SEPARATOR + LOCK_FILE_NAME);
+ this.fs = FSUtils.getFs(this.lockFile.toString(), configuration);
+ }
+
+ @Override
+ public void close() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ fs.delete(this.lockFile, true);
+ } catch (IOException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE), e);
+ }
+ }
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ try {
+ synchronized (LOCK_FILE_NAME) {
+ if (fs.exists(this.lockFile)) {
+ // Check whether lock is already expired or not, if so try to delete lock file
+ if (checkIfExpired()) {
+ fs.delete(this.lockFile, true);
+ }
+ }
+ acquireLock();
+ return fs.exists(this.lockFile);
+ }
+ } catch (IOException | HoodieIOException e) {
+ LOG.info(generateLogStatement(LockState.FAILED_TO_ACQUIRE), e);
+ return false;
+ }
+ }
+
+ @Override
+ public void unlock() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ if (fs.exists(this.lockFile)) {
+ fs.delete(this.lockFile, true);
+ }
+ } catch (IOException io) {
+ throw new HoodieIOException(generateLogStatement(LockState.FAILED_TO_RELEASE), io);
+ }
+ }
+ }
+
+ @Override
+ public String getLock() {
+ return this.lockFile.toString();
+ }
+
+ private boolean checkIfExpired() {
+ if (lockTimeoutMinutes == 0) {
+ return false;
+ }
+ try {
+ long modificationTime = fs.getFileStatus(this.lockFile).getModificationTime();
+ if (System.currentTimeMillis() - modificationTime > lockTimeoutMinutes * 60 * 1000) {
+ return true;
+ }
+ } catch (IOException | HoodieIOException e) {
+ LOG.error(generateLogStatement(LockState.ALREADY_RELEASED) + " failed to get lockFile's modification time", e);
+ }
+ return false;
+ }
+
+ private void acquireLock() {
+ try {
+ fs.create(this.lockFile, false).close();
+ } catch (IOException e) {
+ throw new HoodieIOException(generateLogStatement(LockState.FAILED_TO_ACQUIRE), e);
+ }
+ }
+
+ protected String generateLogStatement(LockState state) {
+ return StringUtils.join(state.name(), " lock at: ", getLock());
+ }
+
+ private void checkRequiredProps(final LockConfiguration config) {
+ ValidationUtils.checkArgument(config.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null) != null
+ || config.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null) != null);
Review Comment:
Here can we just call the `config.getConfig().getString(key)` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1188215602
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
}, {
"hash" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017",
"triggerID" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"triggerType" : "PUSH"
}, {
"hash" : "be5e2b1f70d9a99aae37e2e050121bccc2465f2c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10036",
"triggerID" : "be5e2b1f70d9a99aae37e2e050121bccc2465f2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* be5e2b1f70d9a99aae37e2e050121bccc2465f2c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10036)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186709681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
}, {
"hash" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4d6890001881ab1904785c0ba93b4e2354e1f224 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996)
* 35d61e0415a1b4fa5238f9a7879b1a72e4e408fa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r1061067820
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key())
+ + Path.SEPARATOR + HoodieTableMetaClient.METAFOLDER_NAME;
+ }
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + Path.SEPARATOR + LOCK_FILE_NAME);
+ this.fs = FSUtils.getFs(this.lockFile.toString(), configuration);
+ }
+
+ @Override
+ public void close() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ fs.delete(this.lockFile, true);
+ } catch (IOException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE), e);
+ }
+ }
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ try {
+ synchronized (LOCK_FILE_NAME) {
+ // Check whether lock is already expired, if so try to delete lock file
+ if (fs.exists(this.lockFile) && checkIfExpired()) {
+ fs.delete(this.lockFile, true);
+ }
+ acquireLock();
+ return fs.exists(this.lockFile);
+ }
+ } catch (IOException | HoodieIOException e) {
+ LOG.info(generateLogStatement(LockState.FAILED_TO_ACQUIRE), e);
+ return false;
+ }
+ }
+
+ @Override
+ public void unlock() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ if (fs.exists(this.lockFile)) {
+ fs.delete(this.lockFile, true);
+ }
+ } catch (IOException io) {
+ throw new HoodieIOException(generateLogStatement(LockState.FAILED_TO_RELEASE), io);
+ }
+ }
+ }
+
+ @Override
+ public String getLock() {
+ return this.lockFile.toString();
+ }
+
+ private boolean checkIfExpired() {
+ if (lockTimeoutMinutes == 0) {
+ return false;
+ }
+ try {
+ long modificationTime = fs.getFileStatus(this.lockFile).getModificationTime();
+ if (System.currentTimeMillis() - modificationTime > lockTimeoutMinutes * 60 * 1000) {
+ return true;
+ }
+ } catch (IOException | HoodieIOException e) {
+ LOG.error(generateLogStatement(LockState.ALREADY_RELEASED) + " failed to get lockFile's modification time", e);
+ }
+ return false;
+ }
+
+ private void acquireLock() {
+ try {
+ fs.create(this.lockFile, false).close();
Review Comment:
@fengjian428 :
May I know if you guys tested against both hdfs and S3.
Atleast in S3, not sure if this works across all race conditions.
what if two processes exactly checks fs.exists() concurrently and both detects that file does not exist and goes ahead and creates the file. both creation could succeed. So, how do we ensure only one of them succeed.
@yanghua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1184638409
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4dca5abbeb3dacc65e41ec3c830d6093b3a1b586 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917)
* 47669516e1039019e1b7fa878813b9a88b5b80ae Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1179633359
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b12f205de8b1d4483d05cd54db43f71352ccb493 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a diff in pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r920173062
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static final Logger LOG = LogManager.getLogger(TestFileBasedLockProvider.class);
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @Test
+ public void testAcquireLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ }
+
+ @Test
+ public void testReentrantLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ try {
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ Assertions.fail();
+ } catch (HoodieLockException e) {
+ // expected
+ }
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnlockWithoutLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ fileBasedLockProvider.unlock();
+ }
Review Comment:
Still got conflict error. hudi-common ok hudi-client-common cannot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1183409722
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63568d74644e5ac867f677f68726ad7775896390 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1184624897
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4dca5abbeb3dacc65e41ec3c830d6093b3a1b586 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917)
* 47669516e1039019e1b7fa878813b9a88b5b80ae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1184689395
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47669516e1039019e1b7fa878813b9a88b5b80ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919)
* 8fce774d7b16d44948ad96ae4098d66fde6bb1cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186387982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa8809f397422bbed089749b6ee75de0f9ad9d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991)
* 4d6890001881ab1904785c0ba93b4e2354e1f224 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186396105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4d6890001881ab1904785c0ba93b4e2354e1f224 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r922103787
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int retryMaxCount;
+ private final int retryWaitTimeMs;
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null);
Review Comment:
What I mean is the subpath of the meta table's base path(which should combine hoodie's base path and `HoodieTableMetadata.getMetadataTableBasePath`). CMIIW.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int retryMaxCount;
+ private final int retryWaitTimeMs;
Review Comment:
So what is the meaning of these two variables (and the config items) in your implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1185648458
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8fce774d7b16d44948ad96ae4098d66fde6bb1cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920)
* ef589521e7635f9eb2d015b4c5e9b19d95710528 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r1061068779
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key())
+ + Path.SEPARATOR + HoodieTableMetaClient.METAFOLDER_NAME;
+ }
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + Path.SEPARATOR + LOCK_FILE_NAME);
+ this.fs = FSUtils.getFs(this.lockFile.toString(), configuration);
+ }
+
+ @Override
+ public void close() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ fs.delete(this.lockFile, true);
+ } catch (IOException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE), e);
+ }
+ }
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ try {
+ synchronized (LOCK_FILE_NAME) {
+ // Check whether lock is already expired, if so try to delete lock file
+ if (fs.exists(this.lockFile) && checkIfExpired()) {
+ fs.delete(this.lockFile, true);
+ }
+ acquireLock();
+ return fs.exists(this.lockFile);
+ }
+ } catch (IOException | HoodieIOException e) {
+ LOG.info(generateLogStatement(LockState.FAILED_TO_ACQUIRE), e);
+ return false;
+ }
+ }
+
+ @Override
+ public void unlock() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ if (fs.exists(this.lockFile)) {
+ fs.delete(this.lockFile, true);
+ }
+ } catch (IOException io) {
+ throw new HoodieIOException(generateLogStatement(LockState.FAILED_TO_RELEASE), io);
+ }
+ }
+ }
+
+ @Override
+ public String getLock() {
+ return this.lockFile.toString();
+ }
+
+ private boolean checkIfExpired() {
+ if (lockTimeoutMinutes == 0) {
+ return false;
+ }
+ try {
+ long modificationTime = fs.getFileStatus(this.lockFile).getModificationTime();
+ if (System.currentTimeMillis() - modificationTime > lockTimeoutMinutes * 60 * 1000) {
+ return true;
+ }
+ } catch (IOException | HoodieIOException e) {
+ LOG.error(generateLogStatement(LockState.ALREADY_RELEASED) + " failed to get lockFile's modification time", e);
+ }
+ return false;
+ }
+
+ private void acquireLock() {
+ try {
+ fs.create(this.lockFile, false).close();
Review Comment:
Have left a comment in other PR as well https://github.com/apache/hudi/pull/7440/files#r1061068482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1182446019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b12f205de8b1d4483d05cd54db43f71352ccb493 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804)
* b57aab2953e7a1815096d0cacfd714ba94a411ab UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r920713165
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static final Logger LOG = LogManager.getLogger(TestFileBasedLockProvider.class);
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @Test
+ public void testAcquireLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ }
+
+ @Test
+ public void testReentrantLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ try {
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ Assertions.fail();
+ } catch (HoodieLockException e) {
+ // expected
+ }
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnlockWithoutLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ fileBasedLockProvider.unlock();
+ }
Review Comment:
yeah, if move to `hudi-client-common`, can we implement a multiple writers test case to mock the write conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1184754983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8fce774d7b16d44948ad96ae4098d66fde6bb1cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1182463175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b57aab2953e7a1815096d0cacfd714ba94a411ab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1185694886
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef589521e7635f9eb2d015b4c5e9b19d95710528 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1179580689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b12f205de8b1d4483d05cd54db43f71352ccb493 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1184569612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cae68aeca546c73e9454bb200210d16fdbe6dc7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897)
* 4dca5abbeb3dacc65e41ec3c830d6093b3a1b586 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1184701745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47669516e1039019e1b7fa878813b9a88b5b80ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919)
* 8fce774d7b16d44948ad96ae4098d66fde6bb1cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1184619843
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4dca5abbeb3dacc65e41ec3c830d6093b3a1b586 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1183462734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63568d74644e5ac867f677f68726ad7775896390 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895)
* cae68aeca546c73e9454bb200210d16fdbe6dc7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186815223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
}, {
"hash" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017",
"triggerID" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35d61e0415a1b4fa5238f9a7879b1a72e4e408fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1187380707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
}, {
"hash" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017",
"triggerID" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"triggerType" : "PUSH"
}, {
"hash" : "be5e2b1f70d9a99aae37e2e050121bccc2465f2c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10036",
"triggerID" : "be5e2b1f70d9a99aae37e2e050121bccc2465f2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35d61e0415a1b4fa5238f9a7879b1a72e4e408fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017)
* be5e2b1f70d9a99aae37e2e050121bccc2465f2c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10036)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186197970
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef589521e7635f9eb2d015b4c5e9b19d95710528 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957)
* 5fa8809f397422bbed089749b6ee75de0f9ad9d4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r923337058
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null)
Review Comment:
because FILESYSTEM_LOCK_PATH_PROP_KEY can be null. and getString(key) will throw a exception if key doesn't exist, but I can remove the second one, since there must be one key has value
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileSystemBasedLockProvider.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileSystemBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileSystemBasedLockProvider.class);
+
+ private static final String LOCK_FILE_NAME = "lock";
+
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileSystemBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null);
+ if (StringUtils.isNullOrEmpty(lockDirectory)) {
+ lockDirectory = lockConfiguration.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null)
+ + Path.SEPARATOR + HoodieTableMetaClient.METAFOLDER_NAME;
+ }
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + Path.SEPARATOR + LOCK_FILE_NAME);
+ this.fs = FSUtils.getFs(this.lockFile.toString(), configuration);
+ }
+
+ @Override
+ public void close() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ fs.delete(this.lockFile, true);
+ } catch (IOException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE), e);
+ }
+ }
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ try {
+ synchronized (LOCK_FILE_NAME) {
+ if (fs.exists(this.lockFile)) {
+ // Check whether lock is already expired or not, if so try to delete lock file
+ if (checkIfExpired()) {
+ fs.delete(this.lockFile, true);
+ }
+ }
+ acquireLock();
+ return fs.exists(this.lockFile);
+ }
+ } catch (IOException | HoodieIOException e) {
+ LOG.info(generateLogStatement(LockState.FAILED_TO_ACQUIRE), e);
+ return false;
+ }
+ }
+
+ @Override
+ public void unlock() {
+ synchronized (LOCK_FILE_NAME) {
+ try {
+ if (fs.exists(this.lockFile)) {
+ fs.delete(this.lockFile, true);
+ }
+ } catch (IOException io) {
+ throw new HoodieIOException(generateLogStatement(LockState.FAILED_TO_RELEASE), io);
+ }
+ }
+ }
+
+ @Override
+ public String getLock() {
+ return this.lockFile.toString();
+ }
+
+ private boolean checkIfExpired() {
+ if (lockTimeoutMinutes == 0) {
+ return false;
+ }
+ try {
+ long modificationTime = fs.getFileStatus(this.lockFile).getModificationTime();
+ if (System.currentTimeMillis() - modificationTime > lockTimeoutMinutes * 60 * 1000) {
+ return true;
+ }
+ } catch (IOException | HoodieIOException e) {
+ LOG.error(generateLogStatement(LockState.ALREADY_RELEASED) + " failed to get lockFile's modification time", e);
+ }
+ return false;
+ }
+
+ private void acquireLock() {
+ try {
+ fs.create(this.lockFile, false).close();
+ } catch (IOException e) {
+ throw new HoodieIOException(generateLogStatement(LockState.FAILED_TO_ACQUIRE), e);
+ }
+ }
+
+ protected String generateLogStatement(LockState state) {
+ return StringUtils.join(state.name(), " lock at: ", getLock());
+ }
+
+ private void checkRequiredProps(final LockConfiguration config) {
+ ValidationUtils.checkArgument(config.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY, null) != null
+ || config.getConfig().getString(HoodieWriteConfig.BASE_PATH.key(), null) != null);
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1179580077
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b12f205de8b1d4483d05cd54db43f71352ccb493 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1183520002
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cae68aeca546c73e9454bb200210d16fdbe6dc7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r920716531
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -108,6 +109,12 @@ public class HoodieLockConfig extends HoodieConfig {
.sinceVersion("0.8.0")
Review Comment:
I have an idea, wdyt about setting the `metatable`'s subpath as the default path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1182375308
> @fengjian428 Left some comments you can consider. BTW, did you try to verify this lock mechanism for the `insert override partition` scene?
not yet. I think our users have used this for upsert, insert and drop partition
I can also test this for `insert override partition` scene
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r919605427
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static final Logger LOG = LogManager.getLogger(TestFileBasedLockProvider.class);
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @Test
+ public void testAcquireLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ }
+
+ @Test
+ public void testReentrantLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ try {
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ Assertions.fail();
+ } catch (HoodieLockException e) {
+ // expected
+ }
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnlockWithoutLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ fileBasedLockProvider.unlock();
+ }
Review Comment:
> Just test if can unlock without lock and no exception throws
If you want to judge an exception, you may add a try-catch block and assertion.
> btw, I found MiniDFSCluster cannot be used in Hudi-common-client due to some conflict in pom. can I move UT to Hudi-spark-client?
IMO, I see `MiniDFSCluster` has been used in `hudi-common` e.g. `TestHoodieLogFormat`. Can you try `MiniClusterUtil`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1183399597
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b57aab2953e7a1815096d0cacfd714ba94a411ab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873)
* 63568d74644e5ac867f677f68726ad7775896390 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r920796799
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -108,6 +109,12 @@ public class HoodieLockConfig extends HoodieConfig {
.sinceVersion("0.8.0")
Review Comment:
> I have an idea, wdyt about setting the `metatable`'s subpath as the default path?
yeah, I've thought about it too, if so, since we cannot get basePath from lock config, need to pass extra parameters into the constructor, and other lock providers also need to be refactored. What do you think, is it worth ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1183394855
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b57aab2953e7a1815096d0cacfd714ba94a411ab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873)
* 63568d74644e5ac867f677f68726ad7775896390 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186711279
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
}, {
"hash" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017",
"triggerID" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4d6890001881ab1904785c0ba93b4e2354e1f224 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996)
* 35d61e0415a1b4fa5238f9a7879b1a72e4e408fa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1187368514
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9996",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
}, {
"hash" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017",
"triggerID" : "35d61e0415a1b4fa5238f9a7879b1a72e4e408fa",
"triggerType" : "PUSH"
}, {
"hash" : "be5e2b1f70d9a99aae37e2e050121bccc2465f2c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "be5e2b1f70d9a99aae37e2e050121bccc2465f2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35d61e0415a1b4fa5238f9a7879b1a72e4e408fa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10017)
* be5e2b1f70d9a99aae37e2e050121bccc2465f2c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r917525789
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -108,6 +109,12 @@ public class HoodieLockConfig extends HoodieConfig {
.sinceVersion("0.8.0")
.withDocumentation("For DFS based lock providers, path to store the locks under.");
+ public static final ConfigProperty<Integer> FILESYSTEM_LOCK_EXPIRE = ConfigProperty
+ .key(FILESYSTEM_LOCK_EXPIRE_PROP_KEY)
+ .defaultValue(0)
Review Comment:
If the negative value is illegal, it would be better to clarify it in the documentation.
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static final Logger LOG = LogManager.getLogger(TestFileBasedLockProvider.class);
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
Review Comment:
We need to consider releasing resources like `dfsCluster`.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileBasedLockProvider.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileBasedLockProvider.class);
+
+ private static final String LOCK = "lock";
+
+ private final int retryMaxCount;
+ private final int retryWaitTimeMs;
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ final String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY);
+ this.retryWaitTimeMs = lockConfiguration.getConfig().getInteger(LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY);
+ this.retryMaxCount = lockConfiguration.getConfig().getInteger(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY);
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + "/" + LOCK);
+ this.fs = FSUtils.getFs(this.lockFile.toString(), configuration);
+ }
+
+ @Override
+ public void close() {
+ synchronized (LOCK) {
+ try {
+ fs.delete(this.lockFile, true);
+ } catch (IOException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_RELEASE), e);
+ }
+ }
+ }
+
+ @Override
+ public boolean tryLock(long time, TimeUnit unit) {
+ try {
+ int numRetries = 0;
+ synchronized (LOCK) {
+ while (fs.exists(this.lockFile)) {
+ LOCK.wait(retryWaitTimeMs);
+ numRetries++;
+ if (numRetries > retryMaxCount) {
+ return false;
+ }
+ // Check whether lock is already expired or not, if so try to delete lock file
+ if (lockTimeoutMinutes != 0 && checkIfExpired()) {
+ if (fs.delete(this.lockFile, true)) {
+ break;
+ }
+ }
+ }
+ acquireLock();
+ return fs.exists(this.lockFile);
+ }
+ } catch (IOException | InterruptedException e) {
+ throw new HoodieLockException(generateLogStatement(LockState.FAILED_TO_ACQUIRE), e);
+ }
+ }
+
+ @Override
+ public void unlock() {
+ synchronized (LOCK) {
+ try {
+ if (fs.exists(this.lockFile)) {
+ fs.delete(this.lockFile, true);
+ }
+ } catch (IOException io) {
+ throw new HoodieIOException(generateLogStatement(LockState.FAILED_TO_RELEASE), io);
+ }
+ }
+ }
+
+ @Override
+ public String getLock() {
+ return this.lockFile.toString();
+ }
+
+ private boolean checkIfExpired() {
+ try {
+ long modificationTime = fs.getFileStatus(this.lockFile).getModificationTime();
+ if (System.currentTimeMillis() - modificationTime > lockTimeoutMinutes * 60 * 1000) {
+ return true;
+ }
+ } catch (IOException e) {
+ LOG.error(generateLogStatement(LockState.ALREADY_RELEASED) + " failed to get lockFile's modification time", e);
+ }
+ return false;
+ }
+
+ private void acquireLock() {
+ try {
+ fs.create(this.lockFile, false).close();
+ } catch (IOException e) {
+ throw new HoodieIOException(generateLogStatement(LockState.FAILED_TO_ACQUIRE), e);
+ }
+ }
+
+ protected String generateLogStatement(LockState state) {
+ return StringUtils.join(state.name(), " lock at", getLock());
Review Comment:
`" lock at"` -> `" lock at: "`? looks better?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileBasedLockProvider.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileBasedLockProvider.class);
+
+ private static final String LOCK = "lock";
Review Comment:
`LOCK_FILE_NAME` looks better?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileBasedLockProvider.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileBasedLockProvider implements LockProvider<String>, Serializable {
+
+ private static final Logger LOG = LogManager.getLogger(FileBasedLockProvider.class);
+
+ private static final String LOCK = "lock";
+
+ private final int retryMaxCount;
+ private final int retryWaitTimeMs;
+ private final int lockTimeoutMinutes;
+ private transient FileSystem fs;
+ private transient Path lockFile;
+ protected LockConfiguration lockConfiguration;
+
+ public FileBasedLockProvider(final LockConfiguration lockConfiguration, final Configuration configuration) {
+ checkRequiredProps(lockConfiguration);
+ this.lockConfiguration = lockConfiguration;
+ final String lockDirectory = lockConfiguration.getConfig().getString(FILESYSTEM_LOCK_PATH_PROP_KEY);
+ this.retryWaitTimeMs = lockConfiguration.getConfig().getInteger(LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY);
+ this.retryMaxCount = lockConfiguration.getConfig().getInteger(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY);
+ this.lockTimeoutMinutes = lockConfiguration.getConfig().getInteger(FILESYSTEM_LOCK_EXPIRE_PROP_KEY);
+ this.lockFile = new Path(lockDirectory + "/" + LOCK);
Review Comment:
`/` -> `Path.SEPARATOR` looks better?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static final Logger LOG = LogManager.getLogger(TestFileBasedLockProvider.class);
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @Test
+ public void testAcquireLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
Review Comment:
Shall we assert the result of the `tryLock` method?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestFileBasedLockProvider.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hudi.client.transaction.lock.FileBasedLockProvider;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.testutils.minicluster.HdfsTestService;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY;
+
+public class TestFileBasedLockProvider {
+ private static final Logger LOG = LogManager.getLogger(TestFileBasedLockProvider.class);
+ private static HdfsTestService hdfsTestService;
+ private static MiniDFSCluster dfsCluster;
+ private static LockConfiguration lockConfiguration;
+ private static Configuration hadoopConf;
+
+ @BeforeAll
+ public static void setup() throws IOException {
+ hdfsTestService = new HdfsTestService();
+ dfsCluster = hdfsTestService.start(true);
+ hadoopConf = dfsCluster.getFileSystem().getConf();
+
+ Properties properties = new Properties();
+ properties.setProperty(FILESYSTEM_LOCK_PATH_PROP_KEY, "/tmp/");
+ properties.setProperty(FILESYSTEM_LOCK_EXPIRE_PROP_KEY, "1");
+ properties.setProperty(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY, "1000");
+ properties.setProperty(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY, "3");
+ lockConfiguration = new LockConfiguration(properties);
+ }
+
+ @Test
+ public void testAcquireLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ fileBasedLockProvider.unlock();
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ }
+
+ @Test
+ public void testReentrantLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ Assertions.assertTrue(fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS));
+ try {
+ fileBasedLockProvider.tryLock(lockConfiguration.getConfig()
+ .getLong(LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY), TimeUnit.MILLISECONDS);
+ Assertions.fail();
+ } catch (HoodieLockException e) {
+ // expected
+ }
+ fileBasedLockProvider.unlock();
+ }
+
+ @Test
+ public void testUnlockWithoutLock() {
+ FileBasedLockProvider fileBasedLockProvider = new FileBasedLockProvider(lockConfiguration, hadoopConf);
+ fileBasedLockProvider.unlock();
+ }
Review Comment:
What's the mechanism of measure for this test case?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/FileBasedLockProvider.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction.lock;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.LockConfiguration;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.lock.LockProvider;
+import org.apache.hudi.common.lock.LockState;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_EXPIRE_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.FILESYSTEM_LOCK_PATH_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY;
+import static org.apache.hudi.common.config.LockConfiguration.LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY;
+
+/**
+ * A FileSystem based lock. This {@link LockProvider} implementation allows to lock table operations
+ * using DFS. Users might need to manually clean the Locker's path if writeClient crash and never run again.
+ * NOTE: This only works for DFS with atomic create/delete operation
+ */
+public class FileBasedLockProvider implements LockProvider<String>, Serializable {
Review Comment:
Would you prefer `FileBasedLockProvider ` or `FileSystemBasedLockProvider `?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r921416413
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -108,6 +109,12 @@ public class HoodieLockConfig extends HoodieConfig {
.sinceVersion("0.8.0")
Review Comment:
Thanks for mentioning this. make the meta path as the default path now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1185644470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8fce774d7b16d44948ad96ae4098d66fde6bb1cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920)
* ef589521e7635f9eb2d015b4c5e9b19d95710528 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1186380835
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9917",
"triggerID" : "4dca5abbeb3dacc65e41ec3c830d6093b3a1b586",
"triggerType" : "PUSH"
}, {
"hash" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9919",
"triggerID" : "47669516e1039019e1b7fa878813b9a88b5b80ae",
"triggerType" : "PUSH"
}, {
"hash" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9920",
"triggerID" : "8fce774d7b16d44948ad96ae4098d66fde6bb1cc",
"triggerType" : "PUSH"
}, {
"hash" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9957",
"triggerID" : "ef589521e7635f9eb2d015b4c5e9b19d95710528",
"triggerType" : "PUSH"
}, {
"hash" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991",
"triggerID" : "5fa8809f397422bbed089749b6ee75de0f9ad9d4",
"triggerType" : "PUSH"
}, {
"hash" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4d6890001881ab1904785c0ba93b4e2354e1f224",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa8809f397422bbed089749b6ee75de0f9ad9d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9991)
* 4d6890001881ab1904785c0ba93b4e2354e1f224 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a diff in pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on code in PR #6071:
URL: https://github.com/apache/hudi/pull/6071#discussion_r920857571
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -108,6 +109,12 @@ public class HoodieLockConfig extends HoodieConfig {
.sinceVersion("0.8.0")
Review Comment:
I see the `LockConfiguration` takes all the config items in `HoodieWriteConfig`:
`this.lockConfiguration = new LockConfiguration(writeConfig.getProps());`
in `LockManager`
Maybe we can extract the base path from it？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1182453697
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b12f205de8b1d4483d05cd54db43f71352ccb493 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804)
* b57aab2953e7a1815096d0cacfd714ba94a411ab Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1183466844
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"triggerType" : "PUSH"
}, {
"hash" : "b12f205de8b1d4483d05cd54db43f71352ccb493",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9804",
"triggerID" : "1179632159",
"triggerType" : "MANUAL"
}, {
"hash" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9873",
"triggerID" : "b57aab2953e7a1815096d0cacfd714ba94a411ab",
"triggerType" : "PUSH"
}, {
"hash" : "63568d74644e5ac867f677f68726ad7775896390",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895",
"triggerID" : "63568d74644e5ac867f677f68726ad7775896390",
"triggerType" : "PUSH"
}, {
"hash" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897",
"triggerID" : "cae68aeca546c73e9454bb200210d16fdbe6dc7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63568d74644e5ac867f677f68726ad7775896390 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9895)
* cae68aeca546c73e9454bb200210d16fdbe6dc7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9897)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #6071: [Hudi 4065] create FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #6071:
URL: https://github.com/apache/hudi/pull/6071#issuecomment-1179632159
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #6071: [HUDI-4065] Add FileBasedLockProvider
Posted by GitBox <gi...@apache.org>.
yanghua merged PR #6071:
URL: https://github.com/apache/hudi/pull/6071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org