You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/27 02:42:30 UTC
[GitHub] [hudi] nsivabalan opened a new pull request, #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
nsivabalan opened a new pull request, #7074:
URL: https://github.com/apache/hudi/pull/7074
### Change Logs
Adding support to test spark structured streaming job to integ test. Added a spark submit job for ease of testing w/ EKS as well.
### Impact
We should be able to test spark structured streaming as part of release certification.
### Risk level (write none, low medium or high below)
low.
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1307535260
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c66a4475a5f312ab6241d0f74684197019494c7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #7074:
URL: https://github.com/apache/hudi/pull/7074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1305220612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611",
"triggerID" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"triggerType" : "PUSH"
}, {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51dd40cb8d37331fb7737d707958e26ea57ac5b2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611)
* 3c66a4475a5f312ab6241d0f74684197019494c7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1307177379
@nsivabalan can you look into CI failure pls
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1305492028
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611",
"triggerID" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"triggerType" : "PUSH"
}, {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c66a4475a5f312ab6241d0f74684197019494c7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1308221646
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c66a4475a5f312ab6241d0f74684197019494c7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829)
* 6aae3ad023fa21c0d19662c632139f89a7263e0f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1986911488
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4632f2f00d1d5736ddb0c22c35835f8cb5f54299 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1305189512
addressed comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1307180416
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c66a4475a5f312ab6241d0f74684197019494c7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1986792694
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856",
"triggerID" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"triggerType" : "PUSH"
}, {
"hash" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8071a25549b3df02d24836d1d76ee05fcd4888c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856)
* 4632f2f00d1d5736ddb0c22c35835f8cb5f54299 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1305214352
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611",
"triggerID" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"triggerType" : "PUSH"
}, {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51dd40cb8d37331fb7737d707958e26ea57ac5b2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611)
* 3c66a4475a5f312ab6241d0f74684197019494c7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1308217707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c66a4475a5f312ab6241d0f74684197019494c7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829)
* 6aae3ad023fa21c0d19662c632139f89a7263e0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1986924548
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22859",
"triggerID" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4632f2f00d1d5736ddb0c22c35835f8cb5f54299 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22859)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1292899601
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611",
"triggerID" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51dd40cb8d37331fb7737d707958e26ea57ac5b2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1292896535
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51dd40cb8d37331fb7737d707958e26ea57ac5b2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1986805417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856",
"triggerID" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"triggerType" : "PUSH"
}, {
"hash" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22859",
"triggerID" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4632f2f00d1d5736ddb0c22c35835f8cb5f54299 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22859)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1308623028
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aae3ad023fa21c0d19662c632139f89a7263e0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #7074:
URL: https://github.com/apache/hudi/pull/7074#discussion_r1006933961
##########
hudi-integ-test/src/main/scala/org/apache/hudi/integ/testsuite/streaming/StructuredStreamingSinkTestWriter.scala:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.integ.testsuite.streaming
+
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig.FAIL_ON_TIMELINE_ARCHIVING_ENABLE
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.streaming.StreamingQueryListener.{QueryProgressEvent, QueryStartedEvent, QueryTerminatedEvent}
+import org.apache.spark.sql.streaming.{OutputMode, ProcessingTime, StreamingQueryListener}
+
+object StructuredStreamingSinkTestWriter {
+
+ var validationComplete: Boolean = false;
+
+ def waitUntilCondition(): Unit = {
+ waitUntilCondition(1000 * 60 * 5, 500)
+ }
+
+ def waitUntilCondition(maxWaitTimeMs: Long, intervalTimeMs: Long): Unit = {
+ var waitSoFar: Long = 0;
+ while (waitSoFar < maxWaitTimeMs && !validationComplete) {
+ println("Waiting for " + intervalTimeMs + ". Total wait time " + waitSoFar)
+ Thread.sleep(intervalTimeMs)
+ waitSoFar += intervalTimeMs
+ }
+ }
+
+ def triggerStreaming(spark: SparkSession, tableType: String, inputPath: String, hudiPath: String, hudiCheckpointPath: String,
+ tableName: String, partitionPathField: String, recordKeyField: String,
+ preCombineField: String): Unit = {
+
+ def validate(): Unit = {
+ println("Validation starting")
+ val inputDf = spark.read.format("parquet").load(inputPath)
+ val hudiDf = spark.read.format("hudi").load(hudiPath)
+ inputDf.registerTempTable("inputTbl")
+ hudiDf.registerTempTable("hudiTbl")
+ assert(spark.sql("select count(distinct " + partitionPathField + ", " + recordKeyField + ") from inputTbl").count ==
+ spark.sql("select count(distinct " + partitionPathField + ", " + recordKeyField + ") from hudiTbl").count)
+ validationComplete = true
+ println("Validation complete")
+ }
+
+ def shutdownListener(spark: SparkSession) = new StreamingQueryListener() {
+ override def onQueryStarted(queryStarted: QueryStartedEvent): Unit = {
+ println("Query started: " + queryStarted.id)
Review Comment:
use logger instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1986781329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856",
"triggerID" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8071a25549b3df02d24836d1d76ee05fcd4888c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1986794391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856",
"triggerID" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"triggerType" : "PUSH"
}, {
"hash" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22859",
"triggerID" : "4632f2f00d1d5736ddb0c22c35835f8cb5f54299",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8071a25549b3df02d24836d1d76ee05fcd4888c9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856)
* 4632f2f00d1d5736ddb0c22c35835f8cb5f54299 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22859)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1293067040
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611",
"triggerID" : "51dd40cb8d37331fb7737d707958e26ea57ac5b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51dd40cb8d37331fb7737d707958e26ea57ac5b2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12611)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7074: [HUDI-5101] Adding spark-structured streaming test support via spark-submit job
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1307191794
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c66a4475a5f312ab6241d0f74684197019494c7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1986750576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856",
"triggerID" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aae3ad023fa21c0d19662c632139f89a7263e0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894)
* 8071a25549b3df02d24836d1d76ee05fcd4888c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22856)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5101] Adding spark-structured streaming test support via spark-submit job [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7074:
URL: https://github.com/apache/hudi/pull/7074#issuecomment-1986749220
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12829",
"triggerID" : "3c66a4475a5f312ab6241d0f74684197019494c7",
"triggerType" : "PUSH"
}, {
"hash" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894",
"triggerID" : "6aae3ad023fa21c0d19662c632139f89a7263e0f",
"triggerType" : "PUSH"
}, {
"hash" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8071a25549b3df02d24836d1d76ee05fcd4888c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6aae3ad023fa21c0d19662c632139f89a7263e0f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12894)
* 8071a25549b3df02d24836d1d76ee05fcd4888c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org