You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2023/08/04 10:29:01 UTC
[GitHub] [hudi] danny0405 opened a new pull request, #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
danny0405 opened a new pull request, #9365:
URL: https://github.com/apache/hudi/pull/9365
…d clustering service
### Change Logs
Since Flink enabled MDT now, we must also keep the lock provider for spark offline table services.
### Impact
none.
### Risk level (write none, low medium or high below)
none.
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9365:
URL: https://github.com/apache/hudi/pull/9365#discussion_r1284450589
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java:
##########
@@ -578,6 +580,20 @@ public static HoodieTableMetaClient createMetaClient(
.build();
}
+ public static void addLockOptions(String basePath, TypedProperties props) {
+ if (!props.containsKey(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key())) {
+ HoodieLockConfig lockConfig = HoodieLockConfig.newBuilder()
+ .withLockProvider(FileSystemBasedLockProvider.class)
+ .withLockWaitTimeInMillis(2000L) // 2s
Review Comment:
Probably, the first step is to support a default config option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9365:
URL: https://github.com/apache/hudi/pull/9365#discussion_r1284893578
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java:
##########
@@ -578,6 +580,20 @@ public static HoodieTableMetaClient createMetaClient(
.build();
}
+ public static void addLockOptions(String basePath, TypedProperties props) {
+ if (!props.containsKey(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key())) {
+ HoodieLockConfig lockConfig = HoodieLockConfig.newBuilder()
+ .withLockProvider(FileSystemBasedLockProvider.class)
+ .withLockWaitTimeInMillis(2000L) // 2s
+ .withFileSystemLockExpire(1) // 1 minute
+ .withClientNumRetries(30)
Review Comment:
Use the default values defined in the `ConfigProperty` in `HoodieLockConfig` instead of hard-coding the values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9365:
URL: https://github.com/apache/hudi/pull/9365#discussion_r1284932402
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java:
##########
@@ -578,6 +580,20 @@ public static HoodieTableMetaClient createMetaClient(
.build();
}
+ public static void addLockOptions(String basePath, TypedProperties props) {
+ if (!props.containsKey(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key())) {
+ HoodieLockConfig lockConfig = HoodieLockConfig.newBuilder()
+ .withLockProvider(FileSystemBasedLockProvider.class)
+ .withLockWaitTimeInMillis(2000L) // 2s
+ .withFileSystemLockExpire(1) // 1 minute
+ .withClientNumRetries(30)
Review Comment:
Not sure, I just keep the config same with Flink to make it more friendly to streaming job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on a diff in pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #9365:
URL: https://github.com/apache/hudi/pull/9365#discussion_r1284327253
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java:
##########
@@ -578,6 +580,20 @@ public static HoodieTableMetaClient createMetaClient(
.build();
}
+ public static void addLockOptions(String basePath, TypedProperties props) {
+ if (!props.containsKey(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key())) {
+ HoodieLockConfig lockConfig = HoodieLockConfig.newBuilder()
+ .withLockProvider(FileSystemBasedLockProvider.class)
+ .withLockWaitTimeInMillis(2000L) // 2s
Review Comment:
Could the user only config the option like `hoodie.write.lock.wait_time_ms_between_retry` etc? Then this could follow these configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1665401700
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6ffde7bdc33f53357e131703071c5040793b715 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666340015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6ffde7bdc33f53357e131703071c5040793b715 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072)
* 0aed2250937742799e04e42c511e7fa08a5b92a2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666495945
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
}, {
"hash" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096",
"triggerID" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"triggerType" : "PUSH"
}, {
"hash" : "945d3046756e286e1777dffbf097def561731f7e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19104",
"triggerID" : "945d3046756e286e1777dffbf097def561731f7e",
"triggerType" : "PUSH"
}, {
"hash" : "320ff70e6f0d02b427d9dc868628934f0323fb4e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19109",
"triggerID" : "320ff70e6f0d02b427d9dc868628934f0323fb4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 945d3046756e286e1777dffbf097def561731f7e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19104)
* 320ff70e6f0d02b427d9dc868628934f0323fb4e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19109)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666375333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6ffde7bdc33f53357e131703071c5040793b715 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072)
* 0aed2250937742799e04e42c511e7fa08a5b92a2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093)
* 24a796b7d68ed9b0fced46a0bf297ac1ebf21c13 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666428319
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
}, {
"hash" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096",
"triggerID" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"triggerType" : "PUSH"
}, {
"hash" : "945d3046756e286e1777dffbf097def561731f7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19104",
"triggerID" : "945d3046756e286e1777dffbf097def561731f7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5599728990aa2b2d7e6dae8ef5c54196e94c773e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096)
* 945d3046756e286e1777dffbf097def561731f7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19104)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666577154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
}, {
"hash" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096",
"triggerID" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"triggerType" : "PUSH"
}, {
"hash" : "945d3046756e286e1777dffbf097def561731f7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19104",
"triggerID" : "945d3046756e286e1777dffbf097def561731f7e",
"triggerType" : "PUSH"
}, {
"hash" : "320ff70e6f0d02b427d9dc868628934f0323fb4e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19109",
"triggerID" : "320ff70e6f0d02b427d9dc868628934f0323fb4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 320ff70e6f0d02b427d9dc868628934f0323fb4e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19109)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666699059
Tests have passed: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=19109&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1665414158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6ffde7bdc33f53357e131703071c5040793b715 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666376785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
}, {
"hash" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0aed2250937742799e04e42c511e7fa08a5b92a2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093)
* 24a796b7d68ed9b0fced46a0bf297ac1ebf21c13 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095)
* 5599728990aa2b2d7e6dae8ef5c54196e94c773e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666415439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
}, {
"hash" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096",
"triggerID" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5599728990aa2b2d7e6dae8ef5c54196e94c773e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666344460
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6ffde7bdc33f53357e131703071c5040793b715 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072)
* 0aed2250937742799e04e42c511e7fa08a5b92a2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666426657
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
}, {
"hash" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096",
"triggerID" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"triggerType" : "PUSH"
}, {
"hash" : "945d3046756e286e1777dffbf097def561731f7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "945d3046756e286e1777dffbf097def561731f7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5599728990aa2b2d7e6dae8ef5c54196e94c773e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096)
* 945d3046756e286e1777dffbf097def561731f7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on a diff in pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #9365:
URL: https://github.com/apache/hudi/pull/9365#discussion_r1284327253
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java:
##########
@@ -578,6 +580,20 @@ public static HoodieTableMetaClient createMetaClient(
.build();
}
+ public static void addLockOptions(String basePath, TypedProperties props) {
+ if (!props.containsKey(HoodieLockConfig.LOCK_PROVIDER_CLASS_NAME.key())) {
+ HoodieLockConfig lockConfig = HoodieLockConfig.newBuilder()
+ .withLockProvider(FileSystemBasedLockProvider.class)
+ .withLockWaitTimeInMillis(2000L) // 2s
Review Comment:
Could the user only config the option like `hoodie.write.lock.wait_time_ms_between_retry` etc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9365:
URL: https://github.com/apache/hudi/pull/9365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666385863
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
}, {
"hash" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096",
"triggerID" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24a796b7d68ed9b0fced46a0bf297ac1ebf21c13 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095)
* 5599728990aa2b2d7e6dae8ef5c54196e94c773e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1666494038
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
}, {
"hash" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19093",
"triggerID" : "0aed2250937742799e04e42c511e7fa08a5b92a2",
"triggerType" : "PUSH"
}, {
"hash" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19095",
"triggerID" : "24a796b7d68ed9b0fced46a0bf297ac1ebf21c13",
"triggerType" : "PUSH"
}, {
"hash" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096",
"triggerID" : "5599728990aa2b2d7e6dae8ef5c54196e94c773e",
"triggerType" : "PUSH"
}, {
"hash" : "945d3046756e286e1777dffbf097def561731f7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19104",
"triggerID" : "945d3046756e286e1777dffbf097def561731f7e",
"triggerType" : "PUSH"
}, {
"hash" : "320ff70e6f0d02b427d9dc868628934f0323fb4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "320ff70e6f0d02b427d9dc868628934f0323fb4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5599728990aa2b2d7e6dae8ef5c54196e94c773e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19096)
* 945d3046756e286e1777dffbf097def561731f7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19104)
* 320ff70e6f0d02b427d9dc868628934f0323fb4e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9365: [HUDI-6646] Add default lock provider for spark offline compaction an…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9365:
URL: https://github.com/apache/hudi/pull/9365#issuecomment-1665962704
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d6ffde7bdc33f53357e131703071c5040793b715",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072",
"triggerID" : "d6ffde7bdc33f53357e131703071c5040793b715",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6ffde7bdc33f53357e131703071c5040793b715 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19072)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org