You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/06 11:40:36 UTC
[GitHub] [hudi] djKooks opened a new pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
djKooks opened a new pull request #3608:
URL: https://github.com/apache/hudi/pull/3608
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
HoodieMultiTableDeltaStreamer is just a wrapper on top of HoodieDeltaStreamer and `--enable-sync` parameter needs to be introduced there so that both the streamer classes are in sync.
## Verify this pull request
- Add `enableMetaSync` config for `HoodieMultiTableDeltaStreamer`, to sync with `HoodieDeltaStreamer`
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-917883971
> @pratyakshsharma thanks for help~please share if there's other features needs to be improved/fixed pray
(will keep look-on the project)
Definitely! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma merged pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
pratyakshsharma merged pull request #3608:
URL: https://github.com/apache/hudi/pull/3608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4abba5e932ccab8470572e58de25e3310216c5e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
pratyakshsharma edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-917883971
> @pratyakshsharma thanks for help~please share if there's other features needs to be improved/fixed pray
(will keep look-on the project)
Definitely! You can search for jiras with newbie label and pick them up one by one :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
}, {
"hash" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171",
"triggerID" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"triggerType" : "PUSH"
}, {
"hash" : "9d7898a76a4c7fdc48d20cd6bb800434a3d994e8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d7898a76a4c7fdc48d20cd6bb800434a3d994e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e0f34cb36f6f279d2daede6caa5ac4803f117b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171)
* 9d7898a76a4c7fdc48d20cd6bb800434a3d994e8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-914930154
> @pratyakshsharma there are quite a few requests around the multi delta streamer. Are you still interested in helping improve it and be point person for all feature contribution? please let me know
Big yes to that :) . Please let me know if I can help for anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-914637540
@pratyakshsharma there are quite a few requests around the multi delta streamer. Are you still interested in helping improve it and be point person for all feature contribution? please let me know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-917864154
LGTM. Will merge it once CI passes.
Thank you for your contribution @djKooks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] djKooks commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
djKooks commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-917741626
@pratyakshsharma thanks for comment. Updated 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] djKooks commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
djKooks commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-917881785
@pratyakshsharma thanks for help~please share if there's other features needs to be improved/fixed 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
}, {
"hash" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171",
"triggerID" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"triggerType" : "PUSH"
}, {
"hash" : "9d7898a76a4c7fdc48d20cd6bb800434a3d994e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2172",
"triggerID" : "9d7898a76a4c7fdc48d20cd6bb800434a3d994e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e0f34cb36f6f279d2daede6caa5ac4803f117b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171)
* 9d7898a76a4c7fdc48d20cd6bb800434a3d994e8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2172)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8227a3cacb78456b8f2c350347c6b4ac99cd31d8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4abba5e932ccab8470572e58de25e3310216c5e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048)
* 8227a3cacb78456b8f2c350347c6b4ac99cd31d8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-918200545
@pratyakshsharma This broke master due to a failing test. Please be careful about merging. by ensuring the CI in azure actually passes before you land.
https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=2180&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc&t=e0ae894b-41c9-5f4b-7ed2-bdf5243b02e7
#3654 I am going to land this soon, which I think fixes this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] djKooks edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
djKooks edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-917881785
@pratyakshsharma thanks for help~please share if there's other features needs to be improved/fixed 🙏
(will keep look-on the project)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
}, {
"hash" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8227a3cacb78456b8f2c350347c6b4ac99cd31d8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049)
* 3e0f34cb36f6f279d2daede6caa5ac4803f117b2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
}, {
"hash" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171",
"triggerID" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e0f34cb36f6f279d2daede6caa5ac4803f117b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
}, {
"hash" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171",
"triggerID" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8227a3cacb78456b8f2c350347c6b4ac99cd31d8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049)
* 3e0f34cb36f6f279d2daede6caa5ac4803f117b2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4abba5e932ccab8470572e58de25e3310216c5e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2049",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
}, {
"hash" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2171",
"triggerID" : "3e0f34cb36f6f279d2daede6caa5ac4803f117b2",
"triggerType" : "PUSH"
}, {
"hash" : "9d7898a76a4c7fdc48d20cd6bb800434a3d994e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2172",
"triggerID" : "9d7898a76a4c7fdc48d20cd6bb800434a3d994e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9d7898a76a4c7fdc48d20cd6bb800434a3d994e8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2172)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#discussion_r706587965
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java
##########
@@ -187,7 +188,7 @@ public void testMultiTableExecutionWithParquetSource() throws IOException {
// add only common props. later we can add per table props
String parquetPropsFile = populateCommonPropsAndWriteToFile();
- HoodieMultiTableDeltaStreamer.Config cfg = TestHelpers.getConfig(parquetPropsFile, dfsBasePath + "/config", ParquetDFSSource.class.getName(), false,
+ HoodieMultiTableDeltaStreamer.Config cfg = TestHelpers.getConfig(parquetPropsFile, dfsBasePath + "/config", ParquetDFSSource.class.getName(), false, true,
Review comment:
ditto
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java
##########
@@ -218,7 +219,7 @@ public void testMultiTableExecutionWithParquetSource() throws IOException {
@Test
public void testTableLevelProperties() throws IOException {
- HoodieMultiTableDeltaStreamer.Config cfg = TestHelpers.getConfig(PROPS_FILENAME_TEST_SOURCE1, dfsBasePath + "/config", TestDataSource.class.getName(), false);
+ HoodieMultiTableDeltaStreamer.Config cfg = TestHelpers.getConfig(PROPS_FILENAME_TEST_SOURCE1, dfsBasePath + "/config", TestDataSource.class.getName(), false, true);
Review comment:
ditto
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java
##########
@@ -138,7 +139,7 @@ public void testMultiTableExecutionWithKafkaSource() throws IOException {
testUtils.sendMessages(topicName1, Helpers.jsonifyRecords(dataGenerator.generateInsertsAsPerSchema("000", 5, HoodieTestDataGenerator.TRIP_SCHEMA)));
testUtils.sendMessages(topicName2, Helpers.jsonifyRecords(dataGenerator.generateInsertsAsPerSchema("000", 10, HoodieTestDataGenerator.SHORT_TRIP_SCHEMA)));
- HoodieMultiTableDeltaStreamer.Config cfg = TestHelpers.getConfig(PROPS_FILENAME_TEST_SOURCE1, dfsBasePath + "/config", JsonKafkaSource.class.getName(), false);
+ HoodieMultiTableDeltaStreamer.Config cfg = TestHelpers.getConfig(PROPS_FILENAME_TEST_SOURCE1, dfsBasePath + "/config", JsonKafkaSource.class.getName(), false, true);
Review comment:
Let us keep enableMetaSync as false where enableHiveSync is also false? Otherwise it might lead to confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-914930542
@djKooks Thank you for raising this. Will check this today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4abba5e932ccab8470572e58de25e3310216c5e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048)
* 8227a3cacb78456b8f2c350347c6b4ac99cd31d8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] djKooks commented on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
djKooks commented on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913591891
@pratyakshsharma is `enableMetaSync` just a replacement of `enableHiveSync`, or is there some other changes?
```
if (cfg.enableHiveSync && StringUtils.isNullOrEmpty(tableProperties.getString(DataSourceWriteOptions.HIVE_TABLE().key(), ""))) {
throw new HoodieException("Hive sync table field not provided!");
}
```
=> Is it okay to replace this to `enableMetaSync`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3608: [HUDI-2397] Add `--enable-sync` parameter
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3608:
URL: https://github.com/apache/hudi/pull/3608#issuecomment-913581749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048",
"triggerID" : "d4abba5e932ccab8470572e58de25e3310216c5e",
"triggerType" : "PUSH"
}, {
"hash" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8227a3cacb78456b8f2c350347c6b4ac99cd31d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4abba5e932ccab8470572e58de25e3310216c5e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2048)
* 8227a3cacb78456b8f2c350347c6b4ac99cd31d8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org