You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/12/30 20:45:59 UTC
[PR] use kafka container for tests [hudi]
the-other-tim-brown opened a new pull request, #17758:
URL: https://github.com/apache/hudi/pull/17758
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #17758:
URL: https://github.com/apache/hudi/pull/17758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702297941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ace39b64585671730ea4d54cf70eb24611bf68 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702621507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93d84e7f478706e88b1c5ae935b7267caa0b9136 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711)
* 96c1bd1706f0f7f693b96e92823c70186b9e4435 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712)
* c587632213d67d4269054a771ba8c44dda9d7dd8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702575889
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93d84e7f478706e88b1c5ae935b7267caa0b9136 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711)
* 96c1bd1706f0f7f693b96e92823c70186b9e4435 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712)
* c587632213d67d4269054a771ba8c44dda9d7dd8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3700538275
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1c7b19fa459a103725eafa9a3efd991d227701 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702531745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ace39b64585671730ea4d54cf70eb24611bf68 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708)
* 93d84e7f478706e88b1c5ae935b7267caa0b9136 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711)
* 96c1bd1706f0f7f693b96e92823c70186b9e4435 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712)
* c587632213d67d4269054a771ba8c44dda9d7dd8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702302338
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ace39b64585671730ea4d54cf70eb24611bf68 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708)
* 93d84e7f478706e88b1c5ae935b7267caa0b9136 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702523503
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ace39b64585671730ea4d54cf70eb24611bf68 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708)
* 93d84e7f478706e88b1c5ae935b7267caa0b9136 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711)
* 96c1bd1706f0f7f693b96e92823c70186b9e4435 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3700568670
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1c7b19fa459a103725eafa9a3efd991d227701 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702727237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
}, {
"hash" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716",
"triggerID" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96c1bd1706f0f7f693b96e92823c70186b9e4435 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712)
* c587632213d67d4269054a771ba8c44dda9d7dd8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713)
* c02717e0af1db02b94c75b6c64d49df1cca02776 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702722519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
}, {
"hash" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96c1bd1706f0f7f693b96e92823c70186b9e4435 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712)
* c587632213d67d4269054a771ba8c44dda9d7dd8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713)
* c02717e0af1db02b94c75b6c64d49df1cca02776 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702295727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1c7b19fa459a103725eafa9a3efd991d227701 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690)
* 93ace39b64585671730ea4d54cf70eb24611bf68 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17758:
URL: https://github.com/apache/hudi/pull/17758#discussion_r2655778361
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieMultiTableStreamer.java:
##########
@@ -460,8 +460,10 @@ private static String resetTarget(Config configuration, String database, String
public void sync() {
for (TableExecutionContext context : tableExecutionContexts) {
try {
- new HoodieStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
+ HoodieStreamer streamer = new HoodieStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties()));
+ streamer.sync();
successTables.add(Helpers.getTableWithDatabase(context));
+ streamer.shutdownGracefully();
Review Comment:
Yes, makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702520999
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ace39b64585671730ea4d54cf70eb24611bf68 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708)
* 93d84e7f478706e88b1c5ae935b7267caa0b9136 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711)
* 96c1bd1706f0f7f693b96e92823c70186b9e4435 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702389262
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93ace39b64585671730ea4d54cf70eb24611bf68 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708)
* 93d84e7f478706e88b1c5ae935b7267caa0b9136 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702895935
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
}, {
"hash" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716",
"triggerID" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"triggerType" : "PUSH"
}, {
"hash" : "6dfb53d8622c300d580df924f0931466dbbabb7f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6dfb53d8622c300d580df924f0931466dbbabb7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c02717e0af1db02b94c75b6c64d49df1cca02776 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716)
* 6dfb53d8622c300d580df924f0931466dbbabb7f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3703002909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
}, {
"hash" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716",
"triggerID" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"triggerType" : "PUSH"
}, {
"hash" : "6dfb53d8622c300d580df924f0931466dbbabb7f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10720",
"triggerID" : "6dfb53d8622c300d580df924f0931466dbbabb7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6dfb53d8622c300d580df924f0931466dbbabb7f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10720)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3700540670
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1c7b19fa459a103725eafa9a3efd991d227701 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #17758:
URL: https://github.com/apache/hudi/pull/17758#discussion_r2655763414
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieMultiTableStreamer.java:
##########
@@ -460,8 +460,10 @@ private static String resetTarget(Config configuration, String database, String
public void sync() {
for (TableExecutionContext context : tableExecutionContexts) {
try {
- new HoodieStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
+ HoodieStreamer streamer = new HoodieStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties()));
+ streamer.sync();
successTables.add(Helpers.getTableWithDatabase(context));
+ streamer.shutdownGracefully();
Review Comment:
minor. should we move this to finally block?
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/BaseTestKafkaSource.java:
##########
@@ -71,16 +71,16 @@ public abstract class BaseTestKafkaSource extends SparkClientFunctionalTestHarne
protected final Option<SourceProfileSupplier> sourceProfile = Option.of(mock(SourceProfileSupplier.class));
protected SchemaProvider schemaProvider;
- protected KafkaTestUtils testUtils;
+ protected static KafkaTestUtils testUtils;
- @BeforeEach
- public void initClass() {
+ @BeforeAll
+ public static void setupKafka() {
testUtils = new KafkaTestUtils();
testUtils.setup();
}
- @AfterEach
- public void cleanupClass() {
+ @AfterAll
Review Comment:
should we keep `AfterEach` and add `testUtils.deleteTopics();` in that?
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestAvroKafkaSource.java:
##########
@@ -87,21 +86,17 @@ public class TestAvroKafkaSource extends SparkClientFunctionalTestHarness {
protected SchemaProvider schemaProvider;
- protected KafkaTestUtils testUtils;
+ protected static KafkaTestUtils testUtils;
@BeforeAll
public static void initClass() {
dataGen = new HoodieTestDataGenerator(0xDEED);
- }
-
- @BeforeEach
- public void setup() {
testUtils = new KafkaTestUtils();
testUtils.setup();
}
- @AfterEach
- public void tearDown() {
+ @AfterAll
Review Comment:
should we keep `AfterEach` and add `testUtils.deleteTopics();` in that?
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/debezium/TestAbstractDebeziumSource.java:
##########
@@ -62,28 +60,19 @@ public abstract class TestAbstractDebeziumSource extends UtilitiesTestBase {
private final String testTopicName = "hoodie_test_" + UUID.randomUUID();
private final HoodieIngestionMetrics metrics = mock(HoodieIngestionMetrics.class);
- private KafkaTestUtils testUtils;
+ private static KafkaTestUtils testUtils;
@BeforeAll
public static void initClass() throws Exception {
UtilitiesTestBase.initTestServices();
- }
-
- @BeforeEach
- public void setUpKafkaTestUtils() {
testUtils = new KafkaTestUtils();
testUtils.setup();
}
- @AfterEach
Review Comment:
should we keep `AfterEach` and add `testUtils.deleteTopics();` in that?
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestKafkaOffsetGen.java:
##########
@@ -89,16 +89,16 @@ public class TestKafkaOffsetGen {
private final String testTopicName = "hoodie_test_" + UUID.randomUUID();
private HoodieIngestionMetrics metrics = mock(HoodieIngestionMetrics.class);
- private KafkaTestUtils testUtils;
+ private static KafkaTestUtils testUtils;
- @BeforeEach
- public void setup() throws Exception {
+ @BeforeAll
+ public static void setup() throws Exception {
testUtils = new KafkaTestUtils();
testUtils.setup();
}
- @AfterEach
- public void teardown() throws Exception {
+ @AfterAll
Review Comment:
same comment as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702702735
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96c1bd1706f0f7f693b96e92823c70186b9e4435 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712)
* c587632213d67d4269054a771ba8c44dda9d7dd8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702897823
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
}, {
"hash" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716",
"triggerID" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"triggerType" : "PUSH"
}, {
"hash" : "6dfb53d8622c300d580df924f0931466dbbabb7f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10720",
"triggerID" : "6dfb53d8622c300d580df924f0931466dbbabb7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c02717e0af1db02b94c75b6c64d49df1cca02776 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716)
* 6dfb53d8622c300d580df924f0931466dbbabb7f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10720)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] use kafka container for tests [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17758:
URL: https://github.com/apache/hudi/pull/17758#discussion_r2655671087
##########
hudi-tests-common/pom.xml:
##########
@@ -244,10 +244,5 @@
<artifactId>testcontainers</artifactId>
<scope>compile</scope>
</dependency>
- <dependency>
- <groupId>org.testcontainers</groupId>
Review Comment:
Moved this to be where it is required
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/KafkaTestUtils.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.testutils;
+
+import org.apache.hudi.common.util.StringUtils;
+
+import org.apache.kafka.clients.admin.AdminClient;
+import org.apache.kafka.clients.admin.AdminClientConfig;
+import org.apache.kafka.clients.admin.Config;
+import org.apache.kafka.clients.admin.ConfigEntry;
+import org.apache.kafka.clients.admin.NewTopic;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.TopicExistsException;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.kafka.ConfluentKafkaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+public class KafkaTestUtils {
Review Comment:
Drop in replacement for the existing KafkaTestUtils but does not require zookeeper
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -704,31 +702,31 @@ static HoodieDeltaStreamer.Config makeConfigForHudiIncrSrc(String srcBasePath, S
static void assertAtleastNCompactionCommits(int minExpected, String tablePath) {
HoodieTableMetaClient meta = createMetaClient(storage, tablePath);
HoodieTimeline timeline = meta.getActiveTimeline().getCommitAndReplaceTimeline().filterCompletedInstants();
- LOG.info("Timeline Instants=" + meta.getActiveTimeline().getInstants());
+ LOG.info("Timeline Instants={}", meta.getActiveTimeline().getInstants());
Review Comment:
Fixed logging to avoid the toString calls while I was updating this file
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -140,15 +140,12 @@ public class HoodieDeltaStreamerTestBase extends UtilitiesTestBase {
protected static String topicName;
protected static String defaultSchemaProviderClassName = FilebasedSchemaProvider.class.getName();
protected static int testNum = 1;
-
- Map<String, String> hudiOpts = new HashMap<>();
- public KafkaTestUtils testUtils;
+ protected static KafkaTestUtils testUtils;
Review Comment:
Tests are now only setting up kafka once to cut down on the overhead
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/multisync/TestMultipleMetaSync.java:
##########
@@ -49,7 +49,7 @@ void testMultipleMetaStore() throws Exception {
MockSyncTool2.syncSuccess = false;
// Initial bulk insert to ingest to first hudi table
HoodieDeltaStreamer.Config cfg = getConfig(tableBasePath, getSyncNames("MockSyncTool1", "MockSyncTool2"));
- new HoodieDeltaStreamer(cfg, jsc, fs, hiveServer.getHiveConf()).sync();
+ syncOnce(new HoodieDeltaStreamer(cfg, jsc, fs, hiveServer.getHiveConf()));
Review Comment:
Lots of changes in this PR are using utils or other means to ensure the streamer is shutdown
##########
azure-pipelines-20230430.yml:
##########
@@ -378,6 +378,7 @@ stages:
command: 'run'
arguments: >
-v $(Build.SourcesDirectory):/hudi
+ -v /var/run/docker.sock:/var/run/docker.sock
Review Comment:
This is required for running docker in docker
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieMultiTableStreamer.java:
##########
@@ -460,8 +460,10 @@ private static String resetTarget(Config configuration, String database, String
public void sync() {
for (TableExecutionContext context : tableExecutionContexts) {
try {
- new HoodieStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
+ HoodieStreamer streamer = new HoodieStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties()));
+ streamer.sync();
successTables.add(Helpers.getTableWithDatabase(context));
+ streamer.shutdownGracefully();
Review Comment:
Ensure the instances are shutdown properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702813622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
}, {
"hash" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716",
"triggerID" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c587632213d67d4269054a771ba8c44dda9d7dd8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713)
* c02717e0af1db02b94c75b6c64d49df1cca02776 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #17758:
URL: https://github.com/apache/hudi/pull/17758#discussion_r2655779884
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/BaseTestKafkaSource.java:
##########
@@ -71,16 +71,16 @@ public abstract class BaseTestKafkaSource extends SparkClientFunctionalTestHarne
protected final Option<SourceProfileSupplier> sourceProfile = Option.of(mock(SourceProfileSupplier.class));
protected SchemaProvider schemaProvider;
- protected KafkaTestUtils testUtils;
+ protected static KafkaTestUtils testUtils;
- @BeforeEach
- public void initClass() {
+ @BeforeAll
+ public static void setupKafka() {
testUtils = new KafkaTestUtils();
testUtils.setup();
}
- @AfterEach
- public void cleanupClass() {
+ @AfterAll
Review Comment:
It generally doesn't make any impact since the topics are so small but I can add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(ci): Hudi-utilities test improvements [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17758:
URL: https://github.com/apache/hudi/pull/17758#issuecomment-3702866150
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10690",
"triggerID" : "fd1c7b19fa459a103725eafa9a3efd991d227701",
"triggerType" : "PUSH"
}, {
"hash" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10708",
"triggerID" : "93ace39b64585671730ea4d54cf70eb24611bf68",
"triggerType" : "PUSH"
}, {
"hash" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10711",
"triggerID" : "93d84e7f478706e88b1c5ae935b7267caa0b9136",
"triggerType" : "PUSH"
}, {
"hash" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10712",
"triggerID" : "96c1bd1706f0f7f693b96e92823c70186b9e4435",
"triggerType" : "PUSH"
}, {
"hash" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10713",
"triggerID" : "c587632213d67d4269054a771ba8c44dda9d7dd8",
"triggerType" : "PUSH"
}, {
"hash" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716",
"triggerID" : "c02717e0af1db02b94c75b6c64d49df1cca02776",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c02717e0af1db02b94c75b6c64d49df1cca02776 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=10716)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org