You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "LinMingQiang (via GitHub)" <gi...@apache.org> on 2023/03/31 07:45:25 UTC
[GitHub] [hudi] LinMingQiang opened a new pull request, #8338: [HUDI-5996] Should verify the consistency of bucket num at job sta…
LinMingQiang opened a new pull request, #8338:
URL: https://github.com/apache/hudi/pull/8338
…rtup.
### Change Logs
Users may sometimes modify the bucket num, and the inconsistency of the bucket num will lead to data duplication and make it unavailability. Maybe there are some other parameters that should also be checked before the job starts
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1504726402
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 86a201c099bbc2016d53d987aa481b779464c9c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1550685909
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1134,6 +1136,11 @@ public PropertyBuilder set(Map<String, Object> props) {
return this;
}
+ public PropertyBuilder setHoodieIndexConf(Properties hoodieIndexConf) {
+ this.hoodieIndexConf = hoodieIndexConf;
+ return this;
Review Comment:
Currently the index config is not a table config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1492652408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77646dd05fec450a4c548b8e0d80758471b4cb3c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1502974088
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77646dd05fec450a4c548b8e0d80758471b4cb3c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024)
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1492152984
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014)
* f2b289cd40a8a5ea2e5e2a7c242792e68061e57c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019)
* 77646dd05fec450a4c548b8e0d80758471b4cb3c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1165032150
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
We need to fire another issue first, to discuss promote the bucket number as table config instead of a write config, after that, we can continue with this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-2034400310
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
}, {
"hash" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284",
"triggerID" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"triggerType" : "PUSH"
}, {
"hash" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 8081f81d180126d9c407eac821dbfbd7f5ae28f2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284)
* 3bb52ed2a7193d4cae00f55339b00b17e7f1993b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1492168808
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2b289cd40a8a5ea2e5e2a7c242792e68061e57c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019)
* 77646dd05fec450a4c548b8e0d80758471b4cb3c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Should verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1491651085
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014)
* f2b289cd40a8a5ea2e5e2a7c242792e68061e57c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-2034903046
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
}, {
"hash" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284",
"triggerID" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"triggerType" : "PUSH"
}, {
"hash" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23091",
"triggerID" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"triggerType" : "PUSH"
}, {
"hash" : "fbbfddc71d0aefd947dcb21bb412c12571f357d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23092",
"triggerID" : "fbbfddc71d0aefd947dcb21bb412c12571f357d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* fbbfddc71d0aefd947dcb21bb412c12571f357d2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23092)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1503340225
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 7abfb144f1c76d65cf00115d5bfa9a82b1a28846 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1504964156
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
}, {
"hash" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284",
"triggerID" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 86a201c099bbc2016d53d987aa481b779464c9c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272)
* 8081f81d180126d9c407eac821dbfbd7f5ae28f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-2034755617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
}, {
"hash" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284",
"triggerID" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"triggerType" : "PUSH"
}, {
"hash" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23091",
"triggerID" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"triggerType" : "PUSH"
}, {
"hash" : "fbbfddc71d0aefd947dcb21bb412c12571f357d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23092",
"triggerID" : "fbbfddc71d0aefd947dcb21bb412c12571f357d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 3bb52ed2a7193d4cae00f55339b00b17e7f1993b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23091)
* fbbfddc71d0aefd947dcb21bb412c12571f357d2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23092)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1505570553
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
}, {
"hash" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284",
"triggerID" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 8081f81d180126d9c407eac821dbfbd7f5ae28f2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1504502481
add test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1550685079
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/ITTestDataStreamWrite.java:
##########
@@ -171,6 +178,38 @@ public void testWriteMergeOnReadWithCompaction(String indexType) throws Exceptio
testWriteToHoodie(conf, "mor_write_with_compact", 1, EXPECTED);
}
+ @Test
+ public void testVerifyConsistencyOfBucketNum() throws Exception {
+ String path = tempFile.getAbsolutePath();
+ Configuration conf = TestConfigurations.getDefaultConf(path);
+ conf.setString(FlinkOptions.INDEX_TYPE, "BUCKET");
+ conf.setInteger(FlinkOptions.BUCKET_INDEX_NUM_BUCKETS, 4);
Review Comment:
Maybe we should move this test into `TestStreamWriteOperatorCoordinator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-2034733630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
}, {
"hash" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284",
"triggerID" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"triggerType" : "PUSH"
}, {
"hash" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23091",
"triggerID" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"triggerType" : "PUSH"
}, {
"hash" : "fbbfddc71d0aefd947dcb21bb412c12571f357d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fbbfddc71d0aefd947dcb21bb412c12571f357d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 3bb52ed2a7193d4cae00f55339b00b17e7f1993b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23091)
* fbbfddc71d0aefd947dcb21bb412c12571f357d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1155829734
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
from my perspective, i think it can be promoted.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
From my perspective, i think it can be promoted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1155668406
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
ok, I think it can be promoted from my perspective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Should verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1491524374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1502996686
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77646dd05fec450a4c548b8e0d80758471b4cb3c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024)
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 7abfb144f1c76d65cf00115d5bfa9a82b1a28846 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1155668406
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
ok, I think it can be promoted from my perspective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1155616040
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
We need to discuss whether to promote the bucket num as the table level config instead first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1155515424
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
I have already stored `HoodieIndexConfig` in the table configuration when initializing the table for the first time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1504953218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
}, {
"hash" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 86a201c099bbc2016d53d987aa481b779464c9c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272)
* 8081f81d180126d9c407eac821dbfbd7f5ae28f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Should verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1491533341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Should verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1491641333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014)
* f2b289cd40a8a5ea2e5e2a7c242792e68061e57c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1155050876
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
Did you notice that the tabe can only be initialized only once?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1551589870
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1134,6 +1136,11 @@ public PropertyBuilder set(Map<String, Object> props) {
return this;
}
+ public PropertyBuilder setHoodieIndexConf(Properties hoodieIndexConf) {
+ this.hoodieIndexConf = hoodieIndexConf;
+ return this;
Review Comment:
After the discussion results, I will modify it. Maybe we need to put the bucket-Index configuration into HoodieTableConfig.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1552651830
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1134,6 +1136,11 @@ public PropertyBuilder set(Map<String, Object> props) {
return this;
}
+ public PropertyBuilder setHoodieIndexConf(Properties hoodieIndexConf) {
+ this.hoodieIndexConf = hoodieIndexConf;
+ return this;
Review Comment:
Yeah, I agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1502986171
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77646dd05fec450a4c548b8e0d80758471b4cb3c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024)
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 7abfb144f1c76d65cf00115d5bfa9a82b1a28846 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1504521063
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 7abfb144f1c76d65cf00115d5bfa9a82b1a28846 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253)
* 86a201c099bbc2016d53d987aa481b779464c9c2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-1504528612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 7abfb144f1c76d65cf00115d5bfa9a82b1a28846 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253)
* 86a201c099bbc2016d53d987aa481b779464c9c2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1155123399
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
We just need to verify when the StreamWriteOperatorCoordinator#start or HoodieTableFactory#setupTableOptions. Just ensure that the job cannot be started if the validation fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8338: [HUDI-5996] Verify the consistency of bucket num at job sta…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8338:
URL: https://github.com/apache/hudi/pull/8338#discussion_r1155498288
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -217,18 +218,52 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setCDCEnabled(conf.getBoolean(FlinkOptions.CDC_ENABLED))
.setCDCSupplementalLoggingMode(conf.getString(FlinkOptions.SUPPLEMENTAL_LOGGING_MODE))
.setTimelineLayoutVersion(1)
+ .setHoodieIndexConf(OptionsResolver.getHoodieIndexConf(conf))
.initTable(hadoopConf, basePath);
LOG.info("Table initialized under base path {}", basePath);
return metaClient;
} else {
LOG.info("Table [{}/{}] already exists, no need to initialize the table",
basePath, conf.getString(FlinkOptions.TABLE_NAME));
- return StreamerUtil.createMetaClient(basePath, hadoopConf);
+ HoodieTableMetaClient client = StreamerUtil.createMetaClient(basePath, hadoopConf);
+ validateTableConfig(conf, client.getTableConfig());
Review Comment:
Yeah, I notice that `the bucket num` is not even a table config, so how could we know the preceding num buckets then ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5996] Verify the consistency of bucket num at job sta… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8338:
URL: https://github.com/apache/hudi/pull/8338#issuecomment-2034415412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16014",
"triggerID" : "23fcb4b6e4a5dde55ba7a4723291db3595bc1fd5",
"triggerType" : "PUSH"
}, {
"hash" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16019",
"triggerID" : "f2b289cd40a8a5ea2e5e2a7c242792e68061e57c",
"triggerType" : "PUSH"
}, {
"hash" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16024",
"triggerID" : "77646dd05fec450a4c548b8e0d80758471b4cb3c",
"triggerType" : "PUSH"
}, {
"hash" : "fccdb147c249b08d856819e028986d76603828e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fccdb147c249b08d856819e028986d76603828e9",
"triggerType" : "PUSH"
}, {
"hash" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16253",
"triggerID" : "7abfb144f1c76d65cf00115d5bfa9a82b1a28846",
"triggerType" : "PUSH"
}, {
"hash" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16272",
"triggerID" : "86a201c099bbc2016d53d987aa481b779464c9c2",
"triggerType" : "PUSH"
}, {
"hash" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16284",
"triggerID" : "8081f81d180126d9c407eac821dbfbd7f5ae28f2",
"triggerType" : "PUSH"
}, {
"hash" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23091",
"triggerID" : "3bb52ed2a7193d4cae00f55339b00b17e7f1993b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fccdb147c249b08d856819e028986d76603828e9 UNKNOWN
* 3bb52ed2a7193d4cae00f55339b00b17e7f1993b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23091)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org