You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xicm (via GitHub)" <gi...@apache.org> on 2023/03/09 09:09:29 UTC
[GitHub] [hudi] xicm opened a new pull request, #8143: [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table
xicm opened a new pull request, #8143:
URL: https://github.com/apache/hudi/pull/8143
### Change Logs
lock_key is `hoodie.write.lock.zookeeper.base_path` + partition + '/' + fileId in SimpleTransactionDirectMarkerBasedDetectionStrategy. If the table is a none partition table, the path of znode will be a wrong format.
### Impact
_Describe any public API or user-facing feature change or any performance impact._
none
### Risk level (write none, low medium or high below)
none
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8143:
URL: https://github.com/apache/hudi/pull/8143#issuecomment-1754337249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632",
"triggerID" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"triggerType" : "PUSH"
}, {
"hash" : "6a22bee0577631ab24f55a89519a63c3859f2ead",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20263",
"triggerID" : "6a22bee0577631ab24f55a89519a63c3859f2ead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bccccd1545bc12e727c30a08e689c05fcc59c1a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632)
* 6a22bee0577631ab24f55a89519a63c3859f2ead Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20263)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #8143:
URL: https://github.com/apache/hudi/pull/8143#discussion_r1351340612
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/DirectMarkerTransactionManager.java:
##########
@@ -83,7 +83,7 @@ private static TypedProperties createUpdatedLockProps(
throw new HoodieNotSupportedException("Only Support ZK-based lock for DirectMarkerTransactionManager now.");
}
TypedProperties props = new TypedProperties(writeConfig.getProps());
- props.setProperty(LockConfiguration.ZK_LOCK_KEY_PROP_KEY, partitionPath + "/" + fileId);
+ props.setProperty(LockConfiguration.ZK_LOCK_KEY_PROP_KEY, (null != partitionPath && !partitionPath.isEmpty()) ? partitionPath + "/" + fileId : fileId);
Review Comment:
On deeper look, this change should be fine as currently, the jobs are failing with exceptions when with the setup mentioned in the ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar merged PR #8143:
URL: https://github.com/apache/hudi/pull/8143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8143:
URL: https://github.com/apache/hudi/pull/8143#issuecomment-1754304659
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632",
"triggerID" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"triggerType" : "PUSH"
}, {
"hash" : "6a22bee0577631ab24f55a89519a63c3859f2ead",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6a22bee0577631ab24f55a89519a63c3859f2ead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bccccd1545bc12e727c30a08e689c05fcc59c1a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632)
* 6a22bee0577631ab24f55a89519a63c3859f2ead UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8143: [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8143:
URL: https://github.com/apache/hudi/pull/8143#issuecomment-1461650211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632",
"triggerID" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bccccd1545bc12e727c30a08e689c05fcc59c1a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8143: [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8143:
URL: https://github.com/apache/hudi/pull/8143#issuecomment-1461914734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632",
"triggerID" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bccccd1545bc12e727c30a08e689c05fcc59c1a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8143:
URL: https://github.com/apache/hudi/pull/8143#issuecomment-1754570849
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15632",
"triggerID" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"triggerType" : "PUSH"
}, {
"hash" : "6a22bee0577631ab24f55a89519a63c3859f2ead",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20263",
"triggerID" : "6a22bee0577631ab24f55a89519a63c3859f2ead",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a22bee0577631ab24f55a89519a63c3859f2ead Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20263)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a diff in pull request #8143: [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #8143:
URL: https://github.com/apache/hudi/pull/8143#discussion_r1181169082
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/DirectMarkerTransactionManager.java:
##########
@@ -83,7 +83,7 @@ private static TypedProperties createUpdatedLockProps(
throw new HoodieNotSupportedException("Only Support ZK-based lock for DirectMarkerTransactionManager now.");
}
TypedProperties props = new TypedProperties(writeConfig.getProps());
- props.setProperty(LockConfiguration.ZK_LOCK_KEY_PROP_KEY, partitionPath + "/" + fileId);
+ props.setProperty(LockConfiguration.ZK_LOCK_KEY_PROP_KEY, (null != partitionPath && !partitionPath.isEmpty()) ? partitionPath + "/" + fileId : fileId);
Review Comment:
This change will pose a challenge when upgrading. During upgrade, we would need all writers to be stopped and upgrade to the version containing this change for safe concurrency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #8143:
URL: https://github.com/apache/hudi/pull/8143#issuecomment-1754300940
@xicm : Doing rebase of this PR. Once the test passes, will land this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8143: [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8143:
URL: https://github.com/apache/hudi/pull/8143#issuecomment-1461630070
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0bccccd1545bc12e727c30a08e689c05fcc59c1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bccccd1545bc12e727c30a08e689c05fcc59c1a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #8143:
URL: https://github.com/apache/hudi/pull/8143#issuecomment-1756611038
Code changes looks good. The test failures are due to flaky master which @codope is looking into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org