You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/13 02:16:59 UTC
[GitHub] [hudi] fengjian428 opened a new pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
fengjian428 opened a new pull request #3648:
URL: https://github.com/apache/hudi/pull/3648
https://issues.apache.org/jira/browse/HUDI-2413
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1036537736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
}, {
"hash" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5924",
"triggerID" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e1771de496452d96b4d53f9a6a630d637c6d7c84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851)
* 945d71199b38ea9a8a119028343b1f7bdeea7732 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5924)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-961588383
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034319203
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848)
* e1771de496452d96b4d53f9a6a630d637c6d7c84 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b4450caee1aef2abc7b34b7220583de4e2addfb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173)
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1019652653
@fengjian428 : Do you think you can address the feedback in the next 2 to 3 weeks. We would like to get this in for 0.11.0. If you are occupied, one of us from the community can take it forward. Let us know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034364277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e1771de496452d96b4d53f9a6a630d637c6d7c84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034159037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#discussion_r804443369
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -378,6 +378,10 @@ private boolean onDeltaSyncShutdown(boolean error) {
@Parameter(names = {"--max-retry-count"}, description = "the max retry count if --retry-on-source-failures is enabled")
public Integer maxRetryCount = 3;
+ @Parameter(names = {"--allow-commit-on-no-checkpoint-change"}, description = "allow commits even if checkpoint has not changed before and after fetch data"
+ + "from souce. This might be useful in sources like SqlSource where there is not checkpoint. ")
Review comment:
Should we also add `Not recommended to enable in continuous mode`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-961588383
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1036537736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
}, {
"hash" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5924",
"triggerID" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e1771de496452d96b4d53f9a6a630d637c6d7c84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851)
* 945d71199b38ea9a8a119028343b1f7bdeea7732 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5924)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
codope merged pull request #3648:
URL: https://github.com/apache/hudi/pull/3648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b4450caee1aef2abc7b34b7220583de4e2addfb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun edited a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
dongkelun edited a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-921776941
Hi,I also encountered the same problem in the previous use. I simply wrote the checkpoint as "0". Overall LGTM but I have a question about whether the empty string is repeated with 'formatAdapter.getSource() instanceof SqlSource', and whether it only needs to add a judgment in DeltaSync?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-961588383
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034316785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848)
* e1771de496452d96b4d53f9a6a630d637c6d7c84 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1036535329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
}, {
"hash" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e1771de496452d96b4d53f9a6a630d637c6d7c84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851)
* 945d71199b38ea9a8a119028343b1f7bdeea7732 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1036535329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
}, {
"hash" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e1771de496452d96b4d53f9a6a630d637c6d7c84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851)
* 945d71199b38ea9a8a119028343b1f7bdeea7732 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-927331354
> I think the nicer way would be not have deltastreamer error out (or at-least control this behavior) if there is no checkpoint, but to provide an empty one? This PR adds source specific checks to the `DeltaSync` class which I'd like to avoid.
I remove the exception throw part, instead of add a warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034316785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848)
* e1771de496452d96b4d53f9a6a630d637c6d7c84 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034364277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e1771de496452d96b4d53f9a6a630d637c6d7c84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1036638418
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
}, {
"hash" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5924",
"triggerID" : "945d71199b38ea9a8a119028343b1f7bdeea7732",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 945d71199b38ea9a8a119028343b1f7bdeea7732 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5924)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917957842
@fengjian428 Just a high level question, how is empty string solving the problems caused by null checkpoint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b4450caee1aef2abc7b34b7220583de4e2addfb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-918059066
> @fengjian428 Just a high level question, how is empty string solving the problems caused by null checkpoint?
In DeltaSync line 432, if both resumeCheckpointStr and checkpointStr is null, will judge as no new data, and won't do the next
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-924487255
@codope : can you review this as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034159037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-935186342
@fengjian428 : can you please address the feedback when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b4450caee1aef2abc7b34b7220583de4e2addfb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173)
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b4450caee1aef2abc7b34b7220583de4e2addfb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 edited a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
fengjian428 edited a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-918059066
> @fengjian428 Just a high level question, how is empty string solving the problems caused by null checkpoint?
In DeltaSync line 432, if both resumeCheckpointStr and checkpointStr is null, will judge as no new data, and won't do the next,and in this case , SqlSource is design as one time job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-961588383
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-925750511
Good for @codope to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034319203
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
}, {
"hash" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851",
"triggerID" : "e1771de496452d96b4d53f9a6a630d637c6d7c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848)
* e1771de496452d96b4d53f9a6a630d637c6d7c84 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034110337
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#discussion_r716397052
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestSqlSource.java
##########
@@ -135,6 +135,23 @@ public void testSqlSourceRowFormat() throws IOException {
assertEquals(10000, fetch1AsRows.getBatch().get().count());
}
+ /**
+ * Runs the test scenario of reading data from the source in row format.
+ * Source has no records.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testSqlSourceCheckpoint() throws IOException {
+ props.setProperty(sqlSourceConfig, "select * from test_sql_table where 1=0");
+ sqlSource = new SqlSource(props, jsc, sparkSession, schemaProvider);
+ sourceFormatAdapter = new SourceFormatAdapter(sqlSource);
+
+ InputBatch<Dataset<Row>> fetch1AsRows =
+ sourceFormatAdapter.fetchNewDataInRowFormat(Option.empty(), Long.MAX_VALUE);
+ assertEquals("", fetch1AsRows.getCheckpointForNextBatch());
Review comment:
We can add this assertion to existing `testSqlSourceZeroRecord` instead of adding a new test. Also, let's assert static constant EMPTY_STRING.
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -336,11 +337,11 @@ public void refreshTimeline() throws IOException {
//if previous checkpoint is an empty string, skip resume use Option.empty()
resumeCheckpointStr = Option.of(commitMetadata.getMetadata(CHECKPOINT_KEY));
} else if (commitMetadata.getOperationType() == WriteOperationType.CLUSTER) {
- // incase of CLUSTER commit, no checkpoint will be available in metadata.
+ // in case of CLUSTER commit or sql source, no checkpoint will be available in metadata.
resumeCheckpointStr = Option.empty();
} else if (HoodieTimeline.compareTimestamps(HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
HoodieTimeline.LESSER_THAN, lastCommit.get().getTimestamp())) {
- throw new HoodieDeltaStreamerException(
+ LOG.warn(
Review comment:
This changes the behavior for all the sources. In some cases it might make sense to throw an exception instead of silently failing. Do you think it would be better to introduce a config `hoodie.deltastreamer.fail.on.missing.checkpoint` which is by default true? For SqlSource, users can set to false.
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestSqlSource.java
##########
@@ -135,6 +135,23 @@ public void testSqlSourceRowFormat() throws IOException {
assertEquals(10000, fetch1AsRows.getBatch().get().count());
}
+ /**
+ * Runs the test scenario of reading data from the source in row format.
+ * Source has no records.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testSqlSourceCheckpoint() throws IOException {
Review comment:
It would be good to have a test for SqlSource in TestHoodieDeltaStreamer as well. You can take it up as a follow-up taks if you like. Typically, for every source we want to have a unit test and a deltastreamer test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-917784069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
dongkelun commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-921776941
I also encountered the same problem in the previous use. I simply wrote the checkpoint as "0". And I have a question about whether the empty string is repeated with 'formatAdapter.getSource() instanceof SqlSource', and whether it only needs to add a judgment in DeltaSync?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-935186342
@fengjian428 : can you address the feedback when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034112696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034112696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5848)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034110337
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2173",
"triggerID" : "9b4450caee1aef2abc7b34b7220583de4e2addfb",
"triggerType" : "PUSH"
}, {
"hash" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410",
"triggerID" : "5022f34c93ba19e6ed7c6829f0f98a1e5afeab49",
"triggerType" : "PUSH"
}, {
"hash" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5022f34c93ba19e6ed7c6829f0f98a1e5afeab49 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2410)
* b8b18d50fb3e92ccda1eec83ba8c9d0c2e9b0734 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#issuecomment-1034110666
@codope : this is ready for review now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org