You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/15 21:35:40 UTC
[GitHub] [hudi] yihua opened a new pull request, #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
yihua opened a new pull request, #6955:
URL: https://github.com/apache/hudi/pull/6955
### Change Logs
By default, the default maximum number of instants to fetch in incremental source (`hoodie.deltastreamer.source.hoodieincr.num_instants`) is 1. The checkpoint of the target Hudi table from the incremental ETL lags behind the source if the ingestion runs at a lower frequency than the source or the ingestion job stalls for some time, causing the data freshness issue.
This PR increases default num_instants so that the incremental ETL can fetch more instants to catch up with the source.
### Impact
**Risk level: low**
Only in the cases mentioned above, more instants are fetched for catchup.
### Documentation Update
N/A. The default value is automatically updated on the Hudi website once the website is built for a new release.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1287563869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
}, {
"hash" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac4d33324d8f0ac206711524502a85194b936f7d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227)
* fbaaa2a0e336d1f13a08da416603df9844f90f27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1279844453
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac4d33324d8f0ac206711524502a85194b936f7d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1287599741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
}, {
"hash" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438",
"triggerID" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbaaa2a0e336d1f13a08da416603df9844f90f27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1287564932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
}, {
"hash" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438",
"triggerID" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac4d33324d8f0ac206711524502a85194b936f7d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227)
* fbaaa2a0e336d1f13a08da416603df9844f90f27 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1289472257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
}, {
"hash" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438",
"triggerID" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"triggerType" : "PUSH"
}, {
"hash" : "1dade489cc8baafe3008f0447d35dd2ac54d18cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12537",
"triggerID" : "1dade489cc8baafe3008f0447d35dd2ac54d18cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbaaa2a0e336d1f13a08da416603df9844f90f27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438)
* 1dade489cc8baafe3008f0447d35dd2ac54d18cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12537)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1279873957
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac4d33324d8f0ac206711524502a85194b936f7d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1289901764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
}, {
"hash" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438",
"triggerID" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"triggerType" : "PUSH"
}, {
"hash" : "1dade489cc8baafe3008f0447d35dd2ac54d18cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12537",
"triggerID" : "1dade489cc8baafe3008f0447d35dd2ac54d18cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1dade489cc8baafe3008f0447d35dd2ac54d18cb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12537)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1283457399
@yihua : there are some valid test failures that needs to be fixed.
can you check that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1279843868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac4d33324d8f0ac206711524502a85194b936f7d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6955:
URL: https://github.com/apache/hudi/pull/6955#issuecomment-1289465047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12227",
"triggerID" : "ac4d33324d8f0ac206711524502a85194b936f7d",
"triggerType" : "PUSH"
}, {
"hash" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438",
"triggerID" : "fbaaa2a0e336d1f13a08da416603df9844f90f27",
"triggerType" : "PUSH"
}, {
"hash" : "1dade489cc8baafe3008f0447d35dd2ac54d18cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1dade489cc8baafe3008f0447d35dd2ac54d18cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fbaaa2a0e336d1f13a08da416603df9844f90f27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12438)
* 1dade489cc8baafe3008f0447d35dd2ac54d18cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #6955: [HUDI-5038] Increase default num_instants to fetch for incremental source
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #6955:
URL: https://github.com/apache/hudi/pull/6955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org