You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/20 14:03:48 UTC
[GitHub] [hudi] yesemsanthoshkumar opened a new pull request, #6726: [WIP] Add transformer capability to individual feeds in MultiTableDeltaStreamer
yesemsanthoshkumar opened a new pull request, #6726:
URL: https://github.com/apache/hudi/pull/6726
### Change Logs
Context: https://apache-hudi.slack.com/archives/C4D716NPQ/p1660215517081789
MultiTableDeltastreamer currently supports single transformer class for all of the data being synced. And it can only be enabled or disabled as a whole. There is no support for enabling transformers for a select feed of data or to use different transformers for different feeds. This PR addresses the same.
The same feature is available in schemaprovider class through `hoodie.deltastreamer.schemaprovider.class` property in table level configs.
### Impact
The impact audience are the confined to users of MultiTableDeltaStreamer and that too if they use transformers. And since this is a new feature, things should run as-is even if this change has been incorporated.
**Risk level: none | low | medium | high**
Low
### Contributor's checklist
- [ x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ x] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar merged pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar merged PR #6726:
URL: https://github.com/apache/hudi/pull/6726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6726:
URL: https://github.com/apache/hudi/pull/6726#discussion_r979170345
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java:
##########
@@ -135,6 +135,7 @@ private void populateTableExecutionContextList(TypedProperties properties, Strin
if (cfg.enableMetaSync && StringUtils.isNullOrEmpty(tableProperties.getString(HoodieSyncConfig.META_SYNC_TABLE_NAME.key(), ""))) {
throw new HoodieException("Meta sync table field not provided!");
}
+ populateTransformerProps(cfg, tableProperties);
Review Comment:
Could you add a test for applying different transformers to different tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sydneyhoran commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "sydneyhoran (via GitHub)" <gi...@apache.org>.
sydneyhoran commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1468270289
Just a quick heads up - I ran the latest master MultiTableDeltaStreamer without `hoodie.deltastreamer.transformer.class` config and I got a NullPointerException due to the .split() in line:
```java
List<String> transformerClassNameOverride = Arrays.asList(typedProperties.getString(Constants.TRANSFORMER_CLASS, null).split(","));
```
because my `TRANSFORMER_CLASS` is null. I changed the function to the following with success:
```java
private void populateTransformerProps(HoodieDeltaStreamer.Config cfg, TypedProperties typedProperties) {
String transformerClass = typedProperties.getString(Constants.TRANSFORMER_CLASS, null);
if (transformerClass != null && !transformerClass.trim().isEmpty()) {
List<String> transformerClassNameOverride = Arrays.asList(transformerClass.split(","));
cfg.transformerClassNames = transformerClassNameOverride;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1445441534
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
}, {
"hash" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846",
"triggerID" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"triggerType" : "PUSH"
}, {
"hash" : "c6a245dfaaf6f1bb6cce2d843360ffaa8d042619",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6a245dfaaf6f1bb6cce2d843360ffaa8d042619",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846)
* c6a245dfaaf6f1bb6cce2d843360ffaa8d042619 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1305412647
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
}, {
"hash" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 7d596be965aa1f45d7102b1a640a61f7f1c7301b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230)
* 0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1279873863
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531)
* 7d596be965aa1f45d7102b1a640a61f7f1c7301b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yesemsanthoshkumar commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "yesemsanthoshkumar (via GitHub)" <gi...@apache.org>.
yesemsanthoshkumar commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1469630836
> Thanks @sydneyhoran for catching this issue. @yesemsanthoshkumar : I will go ahead and revert this commit in master. Can you please make the change as suggested and add tests for null case.
Will do it over the weekend.
Thanks @sydneyhoran for catching this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1445460120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
}, {
"hash" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846",
"triggerID" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"triggerType" : "PUSH"
}, {
"hash" : "c6a245dfaaf6f1bb6cce2d843360ffaa8d042619",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15408",
"triggerID" : "c6a245dfaaf6f1bb6cce2d843360ffaa8d042619",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* c6a245dfaaf6f1bb6cce2d843360ffaa8d042619 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15408)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1252446460
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1252718454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1439565478
@yesemsanthoshkumar : Can you rebase the PR for us to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yesemsanthoshkumar commented on a diff in pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
yesemsanthoshkumar commented on code in PR #6726:
URL: https://github.com/apache/hudi/pull/6726#discussion_r985346731
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java:
##########
@@ -135,6 +135,7 @@ private void populateTableExecutionContextList(TypedProperties properties, Strin
if (cfg.enableMetaSync && StringUtils.isNullOrEmpty(tableProperties.getString(HoodieSyncConfig.META_SYNC_TABLE_NAME.key(), ""))) {
throw new HoodieException("Meta sync table field not provided!");
}
+ populateTransformerProps(cfg, tableProperties);
Review Comment:
@yihua Sure. I'm new to this. I'll work over this weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1306191382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
}, {
"hash" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846",
"triggerID" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1445443479
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
}, {
"hash" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846",
"triggerID" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"triggerType" : "PUSH"
}, {
"hash" : "c6a245dfaaf6f1bb6cce2d843360ffaa8d042619",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15408",
"triggerID" : "c6a245dfaaf6f1bb6cce2d843360ffaa8d042619",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846)
* c6a245dfaaf6f1bb6cce2d843360ffaa8d042619 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15408)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1252454781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1279874738
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531)
* 7d596be965aa1f45d7102b1a640a61f7f1c7301b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1279891281
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 7d596be965aa1f45d7102b1a640a61f7f1c7301b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1283551100
@yesemsanthoshkumar : once you have added tests, let us know. and please rebase w/ latest master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1468330690
Thanks @sydneyhoran for catching this issue. @yesemsanthoshkumar : I will go ahead and revert this commit in master. Can you please make the change as suggested and add tests for null case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yesemsanthoshkumar commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by "yesemsanthoshkumar (via GitHub)" <gi...@apache.org>.
yesemsanthoshkumar commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1445669432
@bvaradar Rebased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yesemsanthoshkumar commented on a diff in pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
yesemsanthoshkumar commented on code in PR #6726:
URL: https://github.com/apache/hudi/pull/6726#discussion_r1000163418
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java:
##########
@@ -245,10 +246,13 @@ public void testTableLevelProperties() throws IOException {
case "dummy_table_short_trip":
String tableLevelKeyGeneratorClass = tableExecutionContext.getProperties().getString(DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME().key());
assertEquals(TestHoodieDeltaStreamer.TestTableLevelGenerator.class.getName(), tableLevelKeyGeneratorClass);
+ List<String> transformerClass = tableExecutionContext.getConfig().transformerClassNames;
Review Comment:
I've added these as tests for checking the transformers. Let me know your thoughts on this. I'm not sure if this is the what you expect.
I'm yet to look at the failed builds. And sure. Will rebase to latest master.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java:
##########
@@ -245,10 +246,13 @@ public void testTableLevelProperties() throws IOException {
case "dummy_table_short_trip":
String tableLevelKeyGeneratorClass = tableExecutionContext.getProperties().getString(DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME().key());
assertEquals(TestHoodieDeltaStreamer.TestTableLevelGenerator.class.getName(), tableLevelKeyGeneratorClass);
+ List<String> transformerClass = tableExecutionContext.getConfig().transformerClassNames;
Review Comment:
@nsivabalan I've added these as tests for checking the transformers. Let me know your thoughts on this. I'm not sure if this is the what you expect.
I'm yet to look at the failed builds. And sure. Will rebase to latest master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yesemsanthoshkumar commented on a diff in pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
yesemsanthoshkumar commented on code in PR #6726:
URL: https://github.com/apache/hudi/pull/6726#discussion_r1015529716
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieMultiTableDeltaStreamer.java:
##########
@@ -245,10 +246,13 @@ public void testTableLevelProperties() throws IOException {
case "dummy_table_short_trip":
String tableLevelKeyGeneratorClass = tableExecutionContext.getProperties().getString(DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME().key());
assertEquals(TestHoodieDeltaStreamer.TestTableLevelGenerator.class.getName(), tableLevelKeyGeneratorClass);
+ List<String> transformerClass = tableExecutionContext.getConfig().transformerClassNames;
Review Comment:
@nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1305419055
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
}, {
"hash" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11531",
"triggerID" : "3ef4d2b5608421ab5f12fb550dba4a1ee43b60ad",
"triggerType" : "PUSH"
}, {
"hash" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230",
"triggerID" : "7d596be965aa1f45d7102b1a640a61f7f1c7301b",
"triggerType" : "PUSH"
}, {
"hash" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846",
"triggerID" : "0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
* 7d596be965aa1f45d7102b1a640a61f7f1c7301b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12230)
* 0f2ad047fa81f3e23c1e8a190379fabc81b6a3cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12846)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6726: [HUDI-4630] Add transformer capability to individual feeds in MultiTableDeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6726:
URL: https://github.com/apache/hudi/pull/6726#issuecomment-1252438068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2c5548f6d483ff9c2a190076d9d64dea61610a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2c5548f6d483ff9c2a190076d9d64dea61610a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org