You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/13 09:40:20 UTC
[GitHub] [hudi] lokeshj1703 opened a new pull request, #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
lokeshj1703 opened a new pull request, #7668:
URL: https://github.com/apache/hudi/pull/7668
### Change Logs
At present, Hudi needs an record key and preCombine key to create an Hudi datasets, which puts an restriction on the kinds of datasets we can create using Hudi.
In order to increase the adoption of Hudi file format across all kinds of derived datasets, similar to Parquet/ORC, we need to offer flexibility to users. I understand that record key is used for upsert primitive and we need preCombine key to break the tie and deduplicate, but there are event data and other datasets without any primary key (append only datasets), which can benefit from Hudi since Hudi ecosystem offers other features such as snapshot isolation, indexes, clustering, delta streamer etc., which could be applied to any datasets without record key.
The idea of this proposal is to make both the record key and preCombine key optional to allow variety of new use cases on top of Hudi.
### Impact
None
### Risk level (write none, low medium or high below)
Medium (Added tests)
### Documentation Update
Adds support for KeylessGenerator for insert operations. This ensures user doesn't need to configure record key for inserts in an immutable dataset.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1381639887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2223ac8a3ba5ded440106339e351acca2fa999ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
kazdy commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1387436584
@lokeshj1703 I also noticed that in flink no_precombine is already supported and is guarded behind a config.
Maybe it would be good to do the same in spark ds?
https://github.com/apache/hudi/blob/c9bc03ed8681ab64eea2520f5511464915389c51/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java#L107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1399474089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* a586f6530984a97d73c95bc0ecee6e0807354e28 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321)
* 155d8af889a02e76e4574e569d8bd30449ecf0a4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1382265894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* d5f7f08e368727e161413d5830f400b989fb6ece Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1403025622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
}, {
"hash" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609",
"triggerID" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"triggerType" : "PUSH"
}, {
"hash" : "5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* be4ac65a664851bff13f7df25af2eeddfe3876f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609)
* 5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402781806
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
}, {
"hash" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609",
"triggerID" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602)
* be4ac65a664851bff13f7df25af2eeddfe3876f1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1382679765
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* d5f7f08e368727e161413d5830f400b989fb6ece Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309)
* 4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1381763918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2223ac8a3ba5ded440106339e351acca2fa999ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306)
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* d5f7f08e368727e161413d5830f400b989fb6ece Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1403194063
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
}, {
"hash" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609",
"triggerID" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"triggerType" : "PUSH"
}, {
"hash" : "5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14617",
"triggerID" : "5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1401111987
few high level points
Dis-allow de-dup. If de-dup is enabled (combine before insert), we will fail the write.
Siva: looks ok to me.
Fail if "upsert" is set. With auto generation of record keys, we can't support upsert. I mean, every record is treated as a new record and so doing an index look up causes only unnecessary overhead.
Siva: I feel we can automatically switch to insert. Its an impl detail. We anyways will document that auto generation of record keys is meant to be used only for immutable use-cases. So, rather than failing, I would prefer to auto switch to "insert".
Fail if "hoodie.merge.allow.duplicate.on.inserts" is not enabled so that hudi does not unintentionaly de-dup due to small file handling.
Siva: we should automatically enable this since this is more of an impl detail. I mean, we should not fail if user does not set this.
Fail if someone choose to use MOR table type:- 2 reasons. a: there are no updates and so no point in choosing MOR. b: preCombine is a mandatory field w/ MOR table. but for table w/ auto generated record keys, precombine if not required to be set.
Siva: seems ok.
Fail if preCombine or recordkey field is set.
Siva: seems ok to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7668:
URL: https://github.com/apache/hudi/pull/7668#discussion_r1085999063
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -77,6 +77,28 @@ object HoodieSparkSqlWriter {
private var asyncCompactionTriggerFnDefined: Boolean = false
private var asyncClusteringTriggerFnDefined: Boolean = false
+ def changeOperationToInsertIfRequired(writeOperationType: WriteOperationType, hoodieConfig: HoodieConfig)
Review Comment:
@kazdy : Would you be able to help us w/ a contribution on this. If you take this branch and work on top of it. As of this patch, we have tested auto generation of record keys works for spark-sql (CTAS and create table + inserts ). So, would appreciate if you can put up a fix on the strict/non-strict mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1403220703
We are closing this for now. will put up proper fix later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1381631043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2223ac8a3ba5ded440106339e351acca2fa999ca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #7668:
URL: https://github.com/apache/hudi/pull/7668#discussion_r1085250796
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -185,7 +191,9 @@ object HoodieWriterUtils {
/**
* Detects conflicts between datasourceKeyGen and existing table configuration keyGen
*/
- def validateKeyGeneratorConfig(datasourceKeyGen: String, tableConfig: HoodieConfig): Unit = {
+ def validateAndSetKeyGeneratorConfigs(datasourceKeyGen: String, log: Logger, hoodieConfig: HoodieConfig,
Review Comment:
better to define a separate log for this object than passing from the caller.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -203,6 +211,44 @@ object HoodieWriterUtils {
diffConfigs.insert(0, "\nConfig conflict(key\tcurrent value\texisting value):\n")
throw new HoodieException(diffConfigs.toString.trim)
}
+
+ val parameters = mutable.Map() ++ inputParams
+ if (hoodieConfig.getBoolean(KeyGeneratorOptions.AUTO_GENERATE_RECORD_KEYS)) {
+ val autoGenerateRecordKey = KeyGeneratorOptions.AUTO_GENERATE_RECORD_KEYS.key()
+ if (hoodieConfig.getBoolean(HoodieWriteConfig.COMBINE_BEFORE_INSERT)) {
+ throw new HoodieKeyGeneratorException(s"Config $autoGenerateRecordKey can not be used when " +
+ s"${HoodieWriteConfig.COMBINE_BEFORE_INSERT.key()} is enabled")
+ }
+ if (!hoodieConfig.getBoolean(HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE)) {
+ hoodieConfig.setValue(HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE, "true")
+ parameters += (HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE.key() -> "true")
+ log.warn(s"Enabling config {${HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE.key()}} when $autoGenerateRecordKey is used")
+ }
+ if (hoodieConfig.getString(DataSourceWriteOptions.TABLE_TYPE) == MOR_TABLE_TYPE_OPT_VAL) {
+ throw new HoodieKeyGeneratorException(s"Config ${DataSourceWriteOptions.TABLE_TYPE.key()} should be set to " +
+ s"COW_TABLE_TYPE_OPT_VAL when $autoGenerateRecordKey is used")
+ }
+ // If OPERATION is explicitly set as UPSERT by the user, throw an exception. If user is using default value then
+ // operation is overridden to INSERT_OPERATION_OPT_VAL
+ if (parameters.getOrElse(OPERATION.key(), StringUtils.EMPTY_STRING) == UPSERT_OPERATION_OPT_VAL) {
+ throw new HoodieKeyGeneratorException(s"Config ${OPERATION.key()} should not be set to $UPSERT_OPERATION_OPT_VAL" +
+ s" when $autoGenerateRecordKey is used")
+ } else if (hoodieConfig.getString(OPERATION) == UPSERT_OPERATION_OPT_VAL) {
+ hoodieConfig.setValue(OPERATION.key(), INSERT_OPERATION_OPT_VAL)
+ parameters += (OPERATION.key() -> INSERT_OPERATION_OPT_VAL)
+ log.warn(s"Setting config ${OPERATION.key()} to $INSERT_OPERATION_OPT_VAL when $autoGenerateRecordKey is used")
+ }
+ if (hoodieConfig.getString(DataSourceWriteOptions.PRECOMBINE_FIELD) != DataSourceWriteOptions.PRECOMBINE_FIELD.defaultValue()) {
Review Comment:
Shouldn't we check for precombine field in `parameters`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402490080
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597)
* 5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402914764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
}, {
"hash" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609",
"triggerID" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* be4ac65a664851bff13f7df25af2eeddfe3876f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lokeshj1703 commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1381549648
@nsivabalan Can you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1382692613
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320)
* a586f6530984a97d73c95bc0ecee6e0807354e28 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1382690496
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402369794
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* dbc0a70c376f7a6139a99239e5aaa41227410d54 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587)
* 085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1382716765
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* a586f6530984a97d73c95bc0ecee6e0807354e28 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1399517467
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 155d8af889a02e76e4574e569d8bd30449ecf0a4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1401598936
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 155d8af889a02e76e4574e569d8bd30449ecf0a4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541)
* dbc0a70c376f7a6139a99239e5aaa41227410d54 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1401585506
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 155d8af889a02e76e4574e569d8bd30449ecf0a4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541)
* dbc0a70c376f7a6139a99239e5aaa41227410d54 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1381756073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2223ac8a3ba5ded440106339e351acca2fa999ca Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306)
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* d5f7f08e368727e161413d5830f400b989fb6ece UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402768588
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
URL: https://github.com/apache/hudi/pull/7668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1403029740
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
}, {
"hash" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609",
"triggerID" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"triggerType" : "PUSH"
}, {
"hash" : "5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14617",
"triggerID" : "5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* be4ac65a664851bff13f7df25af2eeddfe3876f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14609)
* 5ff1afe316ae5fd05aa7ff93fe024f4cad944cb6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lokeshj1703 commented on a diff in pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #7668:
URL: https://github.com/apache/hudi/pull/7668#discussion_r1085608448
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -203,6 +211,44 @@ object HoodieWriterUtils {
diffConfigs.insert(0, "\nConfig conflict(key\tcurrent value\texisting value):\n")
throw new HoodieException(diffConfigs.toString.trim)
}
+
+ val parameters = mutable.Map() ++ inputParams
+ if (hoodieConfig.getBoolean(KeyGeneratorOptions.AUTO_GENERATE_RECORD_KEYS)) {
+ val autoGenerateRecordKey = KeyGeneratorOptions.AUTO_GENERATE_RECORD_KEYS.key()
+ if (hoodieConfig.getBoolean(HoodieWriteConfig.COMBINE_BEFORE_INSERT)) {
+ throw new HoodieKeyGeneratorException(s"Config $autoGenerateRecordKey can not be used when " +
+ s"${HoodieWriteConfig.COMBINE_BEFORE_INSERT.key()} is enabled")
+ }
+ if (!hoodieConfig.getBoolean(HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE)) {
+ hoodieConfig.setValue(HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE, "true")
+ parameters += (HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE.key() -> "true")
+ log.warn(s"Enabling config {${HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE.key()}} when $autoGenerateRecordKey is used")
+ }
+ if (hoodieConfig.getString(DataSourceWriteOptions.TABLE_TYPE) == MOR_TABLE_TYPE_OPT_VAL) {
+ throw new HoodieKeyGeneratorException(s"Config ${DataSourceWriteOptions.TABLE_TYPE.key()} should be set to " +
+ s"COW_TABLE_TYPE_OPT_VAL when $autoGenerateRecordKey is used")
+ }
+ // If OPERATION is explicitly set as UPSERT by the user, throw an exception. If user is using default value then
+ // operation is overridden to INSERT_OPERATION_OPT_VAL
+ if (parameters.getOrElse(OPERATION.key(), StringUtils.EMPTY_STRING) == UPSERT_OPERATION_OPT_VAL) {
+ throw new HoodieKeyGeneratorException(s"Config ${OPERATION.key()} should not be set to $UPSERT_OPERATION_OPT_VAL" +
+ s" when $autoGenerateRecordKey is used")
+ } else if (hoodieConfig.getString(OPERATION) == UPSERT_OPERATION_OPT_VAL) {
+ hoodieConfig.setValue(OPERATION.key(), INSERT_OPERATION_OPT_VAL)
+ parameters += (OPERATION.key() -> INSERT_OPERATION_OPT_VAL)
+ log.warn(s"Setting config ${OPERATION.key()} to $INSERT_OPERATION_OPT_VAL when $autoGenerateRecordKey is used")
+ }
+ if (hoodieConfig.getString(DataSourceWriteOptions.PRECOMBINE_FIELD) != DataSourceWriteOptions.PRECOMBINE_FIELD.defaultValue()) {
Review Comment:
Then we will miss the precombine field set in table config for existing tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #7668:
URL: https://github.com/apache/hudi/pull/7668#discussion_r1086228289
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala:
##########
@@ -18,7 +18,7 @@
package org.apache.spark.sql.hudi.command
import org.apache.hudi.exception.HoodieException
-import org.apache.hudi.{HoodieSparkSqlWriter, SparkAdapterSupport}
+import org.apache.hudi.{DataSourceWriteOptions, HoodieSparkSqlWriter, SparkAdapterSupport}
Review Comment:
nit: unused import
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/SimpleKeyGenerator.java:
##########
@@ -126,8 +126,8 @@ private static void validatePartitionPath(String partitionPathField) {
String.format("Single partition-path field is expected; provided (%s)", partitionPathField));
}
- private static void validateRecordKey(String recordKeyField) {
- checkArgument(recordKeyField == null || !recordKeyField.isEmpty(),
+ private static void validateRecordKey(String recordKeyField, boolean isAutoGenerateRecordKeyEnabled) {
+ checkArgument(recordKeyField == null || !recordKeyField.isEmpty() || isAutoGenerateRecordKeyEnabled,
Review Comment:
Should the validation on record key field be:
1. It is not empty.
2. Or, `isAutoGenerateRecordKeyEnabled`.
I am assuming record key field can be null only when `isAutoGenerateRecordKeyEnabled` is true. We can take it as a followup after confirming why null record key field was allowed in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1382691395
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320)
* a586f6530984a97d73c95bc0ecee6e0807354e28 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1381748151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2223ac8a3ba5ded440106339e351acca2fa999ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306)
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1382678782
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* d5f7f08e368727e161413d5830f400b989fb6ece Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309)
* 4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #7668:
URL: https://github.com/apache/hudi/pull/7668#discussion_r1073828099
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -77,6 +77,28 @@ object HoodieSparkSqlWriter {
private var asyncCompactionTriggerFnDefined: Boolean = false
private var asyncClusteringTriggerFnDefined: Boolean = false
+ def changeOperationToInsertIfRequired(writeOperationType: WriteOperationType, hoodieConfig: HoodieConfig)
Review Comment:
handleConfigsForAutoGenerationOfRecordKeys
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -159,7 +164,15 @@ object HoodieWriterUtils {
diffConfigs.append(s"KeyGenerator:\t$datasourceKeyGen\t$tableConfigKeyGen\n")
}
- val datasourcePartitionFields = params.getOrElse(PARTITIONPATH_FIELD.key(), null)
+ val typedProperties = new TypedProperties()
+ typedProperties.putAll(params)
+ val keyGeneratorClass = HoodieSparkKeyGeneratorFactory.getKeyGeneratorClassName(typedProperties)
+ val datasourcePartitionFields = if (keyGeneratorClass == classOf[KeylessKeyGenerator].getName) {
+ val keyGenerator = HoodieSparkKeyGeneratorFactory.createKeyGenerator(typedProperties)
+ SparkKeyGenUtils.getPartitionColumns(keyGenerator, toProperties(params))
Review Comment:
lets revisit this after 5574.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -1179,6 +1177,88 @@ class TestHoodieSparkSqlWriter {
assertTrue(kg2 == classOf[SimpleKeyGenerator].getName)
}
+ @ParameterizedTest
+ @ValueSource(strings = Array("COPY_ON_WRITE", "MERGE_ON_READ"))
+ def testAutoGenerationOfRecordKeysWithCombineBeforeInsert(tableType: String): Unit = {
+ val _spark = spark
+ import _spark.implicits._
+
+ val initialOpts = Map(
+ HoodieWriteConfig.TBL_NAME.key -> hoodieFooTableName,
+ "hoodie.insert.shuffle.parallelism" -> "1",
+ "hoodie.upsert.shuffle.parallelism" -> "1",
+ DataSourceWriteOptions.TABLE_TYPE.key -> tableType,
+ DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key() -> "org.apache.hudi.keygen.KeylessKeyGenerator",
+ DataSourceWriteOptions.OPERATION.key() -> INSERT_OPERATION_OPT_VAL,
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key() -> "partition:simple",
+ HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE.key() -> "false")
+
+ try {
+ // verify exception is thrown when HoodieWriteConfig.COMBINE_BEFORE_INSERT is enabled
+ val tmpDF = Seq((1, "a1", 10, 1000, "2021-10-16")).toDF("id", "name", "value", "ts", "dt")
+ tmpDF.write.format("org.apache.hudi")
+ .options(initialOpts + (HoodieWriteConfig.COMBINE_BEFORE_INSERT.key() -> "true"))
+ .mode(SaveMode.Append)
+ .save(tempBasePath)
+ } catch {
+ case e: HoodieException =>
+ assertTrue(e.getMessage.contains(s"${classOf[KeylessKeyGenerator].getName} can not be used with config enabled " +
+ s"{${HoodieWriteConfig.COMBINE_BEFORE_INSERT.key()}}"), e.getMessage)
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = Array("COPY_ON_WRITE", "MERGE_ON_READ"))
+ def testAutoGenerationOfRecordKeys(tableType: String): Unit = {
+
+ val dataGen = new HoodieTestDataGenerator()
+ val initialOpts = Map(
+ HoodieWriteConfig.TBL_NAME.key -> hoodieFooTableName,
+ "hoodie.insert.shuffle.parallelism" -> "1",
+ "hoodie.upsert.shuffle.parallelism" -> "1",
+ DataSourceWriteOptions.TABLE_TYPE.key -> tableType,
+ DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key() -> "org.apache.hudi.keygen.KeylessKeyGenerator",
+ DataSourceWriteOptions.OPERATION.key() -> UPSERT_OPERATION_OPT_VAL,
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key() -> "partition:simple",
+ HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE.key() -> "false")
+
+ var totalRecs = 0
+ val extraOptsList = List(
+ Map.empty,
+ Map(
+ DataSourceWriteOptions.PRECOMBINE_FIELD.key -> "timestamp",
+ DataSourceWriteOptions.RECORDKEY_FIELD.key -> "_row_key")
+ )
+ for (extraOpt <- extraOptsList) {
+ val opts = initialOpts ++ extraOpt
+ for (x <- 1 to 5) {
+ val instantTime = "00" + x
+ val genRecsList = if (x == 1) {
+ totalRecs += 100 * 2
+ val inserts = dataGen.generateInserts(instantTime, 100)
+ inserts.addAll(inserts)
+ inserts
+ } else {
+ totalRecs += 10
+ dataGen.generateUniqueUpdates(instantTime, 10)
+ }
+ val records = recordsToStrings(genRecsList).toList
+ val inputDF = spark.read.json(spark.sparkContext.parallelize(records, 2))
+ inputDF.write.format("org.apache.hudi")
+ .options(opts)
+ .mode(SaveMode.Append)
+ .save(tempBasePath)
+
+ val snapshotDF = spark.read.format("org.apache.hudi")
+ .load(tempBasePath)
+ assertEquals(totalRecs, snapshotDF.count())
+ }
+ }
+
+ val metaClient = HoodieTableMetaClient.builder().setConf(spark.sparkContext.hadoopConfiguration).setBasePath(tempBasePath).build()
+ assertEquals(metaClient.getTableConfig.getTableType, HoodieTableType.COPY_ON_WRITE)
Review Comment:
can we write tests for exception/non happy paths.
1. if combine.before.insert is set, and try to send duplicates records, all duplicates should be present in snapshot read.
2. verify that preCombine field is not set in tableConfig. ensure tests sets it explicitly.
3. if record key field config is set, it should not throw if keyless is enabled. and try to inject records w/ same record keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #7668: [HUDI-2681] Make hoodie record_key and preCombine_key optional
Posted by GitBox <gi...@apache.org>.
kazdy commented on code in PR #7668:
URL: https://github.com/apache/hudi/pull/7668#discussion_r1070182730
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -77,6 +77,28 @@ object HoodieSparkSqlWriter {
private var asyncCompactionTriggerFnDefined: Boolean = false
private var asyncClusteringTriggerFnDefined: Boolean = false
+ def changeOperationToInsertIfRequired(writeOperationType: WriteOperationType, hoodieConfig: HoodieConfig)
Review Comment:
afaik in this case when changing op from upsert to insert one should also make sure that insert mode (SQL_INSERT_MODE) is set to "upsert", otherwise either duplicate record will be created (non-strict mode) or error will be thrown (strict mode)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lokeshj1703 commented on a diff in pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #7668:
URL: https://github.com/apache/hudi/pull/7668#discussion_r1083431651
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -203,6 +207,34 @@ object HoodieWriterUtils {
diffConfigs.insert(0, "\nConfig conflict(key\tcurrent value\texisting value):\n")
throw new HoodieException(diffConfigs.toString.trim)
}
+
+ if (hoodieConfig.getBoolean(KeyGeneratorOptions.AUTO_GENERATE_RECORD_KEYS)) {
+ val autoGenerateRecordKey = KeyGeneratorOptions.AUTO_GENERATE_RECORD_KEYS.key()
+ if (hoodieConfig.getBoolean(HoodieWriteConfig.COMBINE_BEFORE_INSERT)) {
+ throw new HoodieKeyGeneratorException(s"Config $autoGenerateRecordKey can not be used when " +
+ s"${HoodieWriteConfig.COMBINE_BEFORE_INSERT.key()} is enabled")
+ }
+ if (!hoodieConfig.getBoolean(HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE)) {
+ throw new HoodieKeyGeneratorException(s"Config ${HoodieWriteConfig.MERGE_ALLOW_DUPLICATE_ON_INSERTS_ENABLE.key()} " +
+ s"should be enabled when $autoGenerateRecordKey is used")
+ }
+ if (hoodieConfig.getString(DataSourceWriteOptions.TABLE_TYPE) == MOR_TABLE_TYPE_OPT_VAL) {
+ throw new HoodieKeyGeneratorException(s"Config ${DataSourceWriteOptions.TABLE_TYPE.key()} should be set to " +
+ s"COW_TABLE_TYPE_OPT_VAL when $autoGenerateRecordKey is used")
+ }
+ if (hoodieConfig.getString(OPERATION) == UPSERT_OPERATION_OPT_VAL) {
Review Comment:
Should we change condition to succeed only if insert and bulk insert is enabled? Is it possible for some other operation to execute in this function apart from {insert,upsert and bulk insert}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1399455080
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* a586f6530984a97d73c95bc0ecee6e0807354e28 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321)
* 155d8af889a02e76e4574e569d8bd30449ecf0a4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1401814355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* dbc0a70c376f7a6139a99239e5aaa41227410d54 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402359600
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* dbc0a70c376f7a6139a99239e5aaa41227410d54 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587)
* 085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402481688
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* dbc0a70c376f7a6139a99239e5aaa41227410d54 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587)
* 085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597)
* 5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7668: [HUDI-2681] Some fixes and config validation when auto generation of record keys is enabled
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7668:
URL: https://github.com/apache/hudi/pull/7668#issuecomment-1402775846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14306",
"triggerID" : "2223ac8a3ba5ded440106339e351acca2fa999ca",
"triggerType" : "PUSH"
}, {
"hash" : "b9d8e9c692bab34d6399304f0d438443480782af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d8e9c692bab34d6399304f0d438443480782af",
"triggerType" : "PUSH"
}, {
"hash" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14309",
"triggerID" : "d5f7f08e368727e161413d5830f400b989fb6ece",
"triggerType" : "PUSH"
}, {
"hash" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14320",
"triggerID" : "4366bb4b3b6d46eb31e06bb9a479003ed91ffd2e",
"triggerType" : "PUSH"
}, {
"hash" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14321",
"triggerID" : "a586f6530984a97d73c95bc0ecee6e0807354e28",
"triggerType" : "PUSH"
}, {
"hash" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14541",
"triggerID" : "155d8af889a02e76e4574e569d8bd30449ecf0a4",
"triggerType" : "PUSH"
}, {
"hash" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14587",
"triggerID" : "dbc0a70c376f7a6139a99239e5aaa41227410d54",
"triggerType" : "PUSH"
}, {
"hash" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14597",
"triggerID" : "085d17a9fbe96bdd6f4fbebe6d95d2afc13d0885",
"triggerType" : "PUSH"
}, {
"hash" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602",
"triggerID" : "5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed",
"triggerType" : "PUSH"
}, {
"hash" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "be4ac65a664851bff13f7df25af2eeddfe3876f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9d8e9c692bab34d6399304f0d438443480782af UNKNOWN
* 5a88ce8a03cbdd3037c327d063511f7e5b9ec5ed Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14602)
* be4ac65a664851bff13f7df25af2eeddfe3876f1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org