You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "harsh1231 (via GitHub)" <gi...@apache.org> on 2023/04/20 17:36:52 UTC
[GitHub] [hudi] harsh1231 opened a new pull request, #8520: [HUDI-6115] Hardening expectation around corruptRecordColumn in chainedtransformer.
harsh1231 opened a new pull request, #8520:
URL: https://github.com/apache/hudi/pull/8520
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1549522553
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ace27d8182562b768d28ccfe34dfc9611706b7d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114)
* f540c4e6eac618acfa15c5ff03d53beac3461341 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1175141306
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
Review Comment:
changed implementation to use separate chainedTransformer for error table .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1531204878
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 56550641906350ac03288ad2f23fff15ade2e9da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684)
* 3cd1d7cad018c67c90ba539d48813d94ac1d1c35 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1172912945
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +52,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isQuarantineEnabled = properties.getBoolean(QUARANTINE_TABLE_ENABLED.key(), QUARANTINE_TABLE_ENABLED.defaultValue());
+ if (isQuarantineEnabled && !isCorruptRecordPresent(dataset)) {
+ dataset = dataset.withColumn(QUARANTINE_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isQuarantineEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on corruptRecordColumn when quarantine enabled/disabled configs are set.
+ * @param dataset
+ * @param isQuarantineEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isQuarantineEnabled) {
+ boolean isCorruptRecordColumnPresent = isCorruptRecordPresent(dataset);
+ if (isQuarantineEnabled != isCorruptRecordColumnPresent) {
Review Comment:
should we consider below condition as validation violation ?
if isQuarantineEnabled = false and isCorruptRecordColumnPresent = true
if there is user schema column QUARANTINE_TABLE_CURRUPT_RECORD_COL_NAME , this will fail , although user is not using quarantine settings .
@rmahindra123 @xushiyan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1523341774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aea97f3976876ed811329b4e7892b99ae97e0bab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669)
* 56550641906350ac03288ad2f23fff15ade2e9da UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1552754937
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
}, {
"hash" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143",
"triggerID" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"triggerType" : "PUSH"
}, {
"hash" : "0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143)
* 0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1531414885
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3cd1d7cad018c67c90ba539d48813d94ac1d1c35 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1550856132
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
}, {
"hash" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143",
"triggerID" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f540c4e6eac618acfa15c5ff03d53beac3461341 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123)
* 79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1552765049
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
}, {
"hash" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143",
"triggerID" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"triggerType" : "PUSH"
}, {
"hash" : "0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17185",
"triggerID" : "0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143)
* 0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1173171771
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled/disabled configs are set.
+ * @param dataset
+ * @param isErrorTableEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isErrorTableEnabled) {
+ boolean isErrorRecordColumnPresent = isErrorRecordPresent(dataset);
+ if (isErrorTableEnabled != isErrorRecordColumnPresent) {
Review Comment:
but whats the next step user can do ? we can't even restart the pipeline right. so, only other option is to disable quarantine.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled/disabled configs are set.
+ * @param dataset
+ * @param isErrorTableEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isErrorTableEnabled) {
+ boolean isErrorRecordColumnPresent = isErrorRecordPresent(dataset);
+ if (isErrorTableEnabled != isErrorRecordColumnPresent) {
Review Comment:
yes, before adding the error record for the first time, we might need to do this validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1172961644
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled/disabled configs are set.
+ * @param dataset
+ * @param isErrorTableEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isErrorTableEnabled) {
+ boolean isErrorRecordColumnPresent = isErrorRecordPresent(dataset);
+ if (isErrorTableEnabled != isErrorRecordColumnPresent) {
Review Comment:
should we consider below condition as validation violation ?
if isErrorTableEnabled = false and isErrorRecordColumnPresent = true
if there is existing schema column ERROR_TABLE_CURRUPT_RECORD_COL_NAME this will fail ( if error table is disabled -> does this condition makes sense )
@rmahindra123 @xushiyan
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled/disabled configs are set.
+ * @param dataset
+ * @param isErrorTableEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isErrorTableEnabled) {
+ boolean isErrorRecordColumnPresent = isErrorRecordPresent(dataset);
+ if (isErrorTableEnabled != isErrorRecordColumnPresent) {
Review Comment:
should we consider below condition as validation violation ?
if isErrorTableEnabled = false and isErrorRecordColumnPresent = true
if there is existing schema column `ERROR_TABLE_CURRUPT_RECORD_COL_NAME` this will fail ( if error table is disabled -> does this condition makes sense )
@rmahindra123 @xushiyan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1173318675
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
Review Comment:
if using error table requires some column to be present, we should enforce this rule within error table itself. Transformer should not need to know about whether error table enabled or not. There could be a cleaner solution to solve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1173318675
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
Review Comment:
if using error table requires some column to be present, we should enforce this rule within error table's related classes. Transformer should not need to know about whether error table enabled or not. There could be a cleaner solution to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1523256074
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aea97f3976876ed811329b4e7892b99ae97e0bab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1175141987
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled/disabled configs are set.
+ * @param dataset
+ * @param isErrorTableEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isErrorTableEnabled) {
+ boolean isErrorRecordColumnPresent = isErrorRecordPresent(dataset);
+ if (isErrorTableEnabled != isErrorRecordColumnPresent) {
Review Comment:
As flows are separate now , above condition is not required to be tested .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1175141987
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled/disabled configs are set.
+ * @param dataset
+ * @param isErrorTableEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isErrorTableEnabled) {
+ boolean isErrorRecordColumnPresent = isErrorRecordPresent(dataset);
+ if (isErrorTableEnabled != isErrorRecordColumnPresent) {
Review Comment:
As flows are seperate now , above condition is not required to be tested .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 closed pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 closed pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
URL: https://github.com/apache/hudi/pull/8520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1187019450
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -63,4 +70,27 @@ public static HoodieErrorTableConfig.ErrorWriteFailureStrategy getErrorWriteFail
String writeFailureStrategy = props.getString(ERROR_TABLE_WRITE_FAILURE_STRATEGY.key());
return HoodieErrorTableConfig.ErrorWriteFailureStrategy.valueOf(writeFailureStrategy);
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled configs are set.
+ * @param dataset
+ */
+ public static void validate(Dataset<Row> dataset) {
+ if (!isErrorTableCorruptRecordColumnPresent(dataset)) {
+ throw new HoodieValidationException(String.format("Invalid condition, columnName=%s "
+ + "is not present in transformer " + "output schema", ERROR_TABLE_CURRUPT_RECORD_COL_NAME));
+ }
+ }
+
+ public static Dataset<Row> addNullValueErrorTableCorruptRecordColumn(Dataset<Row> dataset) {
+ if (!isErrorTableCorruptRecordColumnPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
+ return dataset;
+ }
+
+ private static boolean isErrorTableCorruptRecordColumnPresent(Dataset<Row> dataset) {
+ return Arrays.stream(dataset.columns()).collect(Collectors.toList())
Review Comment:
df.columns is `String[]` , does not have contains method .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1531188756
> @harsh1231 The newly added tests are failing. Can you please check? https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=16684&view=logs&j=9273dbcb-a208-5cd0-5541-8b9925cb3da0&t=3db7c4d5-9ba1-5c0b-fe5f-44ac24504451
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1516815799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f42d020cc32a23c69e72f6be0f3f5b92d54f0b99 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508)
* 6a7c71742f4b8fdc416b885edd0bb3e63687a715 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1520033679
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a7c71742f4b8fdc416b885edd0bb3e63687a715 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509)
* 5ca6c38df3f6511f072a1c65ac291249f0957311 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1187655710
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -28,32 +28,39 @@
import org.apache.hudi.exception.HoodieException;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.exception.HoodieValidationException;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
+import static org.apache.spark.sql.functions.lit;
+
import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;
+import java.util.stream.Collectors;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_CLASS;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_FAILURE_STRATEGY;
+import static org.apache.hudi.utilities.deltastreamer.BaseErrorTableWriter.ERROR_TABLE_CURRUPT_RECORD_COL_NAME;
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1522801441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ca6c38df3f6511f072a1c65ac291249f0957311 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608)
* aea97f3976876ed811329b4e7892b99ae97e0bab Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1537727478
> @harsh1231 The newly added tests are failing. Can you please check? https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=16684&view=logs&j=9273dbcb-a208-5cd0-5541-8b9925cb3da0&t=3db7c4d5-9ba1-5c0b-fe5f-44ac24504451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1529944380
> @harsh1231 The newly added tests are failing. Can you please check? https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=16684&view=logs&j=9273dbcb-a208-5cd0-5541-8b9925cb3da0&t=3db7c4d5-9ba1-5c0b-fe5f-44ac24504451
checking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1187029695
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -28,32 +28,39 @@
import org.apache.hudi.exception.HoodieException;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.exception.HoodieValidationException;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
+import static org.apache.spark.sql.functions.lit;
+
import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;
+import java.util.stream.Collectors;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_CLASS;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_FAILURE_STRATEGY;
+import static org.apache.hudi.utilities.deltastreamer.BaseErrorTableWriter.ERROR_TABLE_CURRUPT_RECORD_COL_NAME;
Review Comment:
So is my understanding correct that this column is never written to storage irrespective of whether it is an incremental source or something else? It is only being used for internal error table validation but never written.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -28,32 +28,39 @@
import org.apache.hudi.exception.HoodieException;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.exception.HoodieValidationException;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
+import static org.apache.spark.sql.functions.lit;
+
import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;
+import java.util.stream.Collectors;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_CLASS;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_FAILURE_STRATEGY;
+import static org.apache.hudi.utilities.deltastreamer.BaseErrorTableWriter.ERROR_TABLE_CURRUPT_RECORD_COL_NAME;
public final class ErrorTableUtils {
-
public static Option<BaseErrorTableWriter> getErrorTableWriter(HoodieDeltaStreamer.Config cfg, SparkSession sparkSession,
- TypedProperties props, JavaSparkContext jssc, FileSystem fs) {
+ TypedProperties props, JavaSparkContext jssc, FileSystem fs) {
String errorTableWriterClass = props.getString(ERROR_TABLE_WRITE_CLASS.key());
ValidationUtils.checkState(!StringUtils.isNullOrEmpty(errorTableWriterClass),
"Missing error table config " + ERROR_TABLE_WRITE_CLASS);
- Class<?>[] argClassArr = new Class[] {HoodieDeltaStreamer.Config.class,
+ Class<?>[] argClassArr = new Class[]{HoodieDeltaStreamer.Config.class,
SparkSession.class, TypedProperties.class, JavaSparkContext.class, FileSystem.class};
String errMsg = "Unable to instantiate ErrorTableWriter with arguments type " + Arrays.toString(argClassArr);
ValidationUtils.checkArgument(ReflectionUtils.hasConstructor(BaseErrorTableWriter.class.getName(), argClassArr), errMsg);
try {
return Option.of((BaseErrorTableWriter) ReflectionUtils.getClass(errorTableWriterClass).getConstructor(argClassArr)
.newInstance(cfg, sparkSession, props, jssc, fs));
- } catch (NoSuchMethodException | InvocationTargetException | InstantiationException | IllegalAccessException e) {
+ } catch (NoSuchMethodException | InvocationTargetException | InstantiationException
+ | IllegalAccessException e) {
Review Comment:
nit: keep all in a single line or have one exception per line for better readability.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -63,4 +70,27 @@ public static HoodieErrorTableConfig.ErrorWriteFailureStrategy getErrorWriteFail
String writeFailureStrategy = props.getString(ERROR_TABLE_WRITE_FAILURE_STRATEGY.key());
return HoodieErrorTableConfig.ErrorWriteFailureStrategy.valueOf(writeFailureStrategy);
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled configs are set.
+ * @param dataset
+ */
+ public static void validate(Dataset<Row> dataset) {
+ if (!isErrorTableCorruptRecordColumnPresent(dataset)) {
+ throw new HoodieValidationException(String.format("Invalid condition, columnName=%s "
+ + "is not present in transformer " + "output schema", ERROR_TABLE_CURRUPT_RECORD_COL_NAME));
+ }
+ }
+
+ public static Dataset<Row> addNullValueErrorTableCorruptRecordColumn(Dataset<Row> dataset) {
+ if (!isErrorTableCorruptRecordColumnPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
+ return dataset;
+ }
+
+ private static boolean isErrorTableCorruptRecordColumnPresent(Dataset<Row> dataset) {
+ return Arrays.stream(dataset.columns()).collect(Collectors.toList())
Review Comment:
I guess the intention is to avoid `collect`? Can we simply do `Arrays.stream(dataset.columns()).anyMatch(col -> col.equalsIgnoreCase(ERROR_TABLE_CURRUPT_RECORD_COL_NAME))`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1549166895
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3cd1d7cad018c67c90ba539d48813d94ac1d1c35 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788)
* 4ace27d8182562b768d28ccfe34dfc9611706b7d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1516735534
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f42d020cc32a23c69e72f6be0f3f5b92d54f0b99 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1516805218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f42d020cc32a23c69e72f6be0f3f5b92d54f0b99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508)
* 6a7c71742f4b8fdc416b885edd0bb3e63687a715 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1531257110
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 56550641906350ac03288ad2f23fff15ade2e9da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684)
* 3cd1d7cad018c67c90ba539d48813d94ac1d1c35 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1550846684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
}, {
"hash" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f540c4e6eac618acfa15c5ff03d53beac3461341 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123)
* 79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1516746955
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f42d020cc32a23c69e72f6be0f3f5b92d54f0b99 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1172962923
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled/disabled configs are set.
+ * @param dataset
+ * @param isErrorTableEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isErrorTableEnabled) {
+ boolean isErrorRecordColumnPresent = isErrorRecordPresent(dataset);
+ if (isErrorTableEnabled != isErrorRecordColumnPresent) {
Review Comment:
@nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1522768626
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ca6c38df3f6511f072a1c65ac291249f0957311 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608)
* aea97f3976876ed811329b4e7892b99ae97e0bab UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1520047075
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a7c71742f4b8fdc416b885edd0bb3e63687a715 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509)
* 5ca6c38df3f6511f072a1c65ac291249f0957311 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1520543439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ca6c38df3f6511f072a1c65ac291249f0957311 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1550199538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f540c4e6eac618acfa15c5ff03d53beac3461341 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1549177568
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3cd1d7cad018c67c90ba539d48813d94ac1d1c35 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788)
* 4ace27d8182562b768d28ccfe34dfc9611706b7d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1172912945
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +52,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isQuarantineEnabled = properties.getBoolean(QUARANTINE_TABLE_ENABLED.key(), QUARANTINE_TABLE_ENABLED.defaultValue());
+ if (isQuarantineEnabled && !isCorruptRecordPresent(dataset)) {
+ dataset = dataset.withColumn(QUARANTINE_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isQuarantineEnabled);
}
return dataset;
}
+
+ /**
+ * validates for constraints on corruptRecordColumn when quarantine enabled/disabled configs are set.
+ * @param dataset
+ * @param isQuarantineEnabled
+ */
+ private void validate(Dataset<Row> dataset, boolean isQuarantineEnabled) {
+ boolean isCorruptRecordColumnPresent = isCorruptRecordPresent(dataset);
+ if (isQuarantineEnabled != isCorruptRecordColumnPresent) {
Review Comment:
should we consider below condition as validation violation ?
if isQuarantineEnabled = false and isCorruptRecordColumnPresent = true
if there is user schema column QUARANTINE_TABLE_CURRUPT_RECORD_COL_NAME , this will fail , although user is not using quarantine settings .
@rmahindra123 @xushiyan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1173318675
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedTransformer.java:
##########
@@ -46,9 +53,33 @@ public List<String> getTransformersNames() {
@Override
public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties) {
Dataset<Row> dataset = rowDataset;
+ boolean isErrorTableEnabled = properties.getBoolean(ERROR_TABLE_ENABLED.key(), ERROR_TABLE_ENABLED.defaultValue());
+ if (isErrorTableEnabled && !isErrorRecordPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
for (Transformer t : transformers) {
dataset = t.apply(jsc, sparkSession, dataset, properties);
+ // validate in every stage to ensure it's not dropped by one of the transformer and added by next transformer.
+ validate(dataset, isErrorTableEnabled);
Review Comment:
if using error table requires some column to be present, we should enforce this rule within error table's related classes. Transformer should not need to know about whether error table enabled or not. There could be a cleaner solution to solve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1182901309
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -28,32 +28,39 @@
import org.apache.hudi.exception.HoodieException;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.exception.HoodieValidationException;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
+import static org.apache.spark.sql.functions.lit;
+
import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;
+import java.util.stream.Collectors;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_CLASS;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_FAILURE_STRATEGY;
+import static org.apache.hudi.utilities.deltastreamer.BaseErrorTableWriter.ERROR_TABLE_CURRUPT_RECORD_COL_NAME;
Review Comment:
So this is like a reserved column for any hudi table then?
should we check if incoming dataset has this column and fail may be? what in case we have a data column w/ the same name?
how happens incase of incremental consumer? it might have this column as a data column right ?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -63,4 +70,27 @@ public static HoodieErrorTableConfig.ErrorWriteFailureStrategy getErrorWriteFail
String writeFailureStrategy = props.getString(ERROR_TABLE_WRITE_FAILURE_STRATEGY.key());
return HoodieErrorTableConfig.ErrorWriteFailureStrategy.valueOf(writeFailureStrategy);
}
+
+ /**
+ * validates for constraints on ErrorRecordColumn when ErrorTable enabled configs are set.
+ * @param dataset
+ */
+ public static void validate(Dataset<Row> dataset) {
+ if (!isErrorTableCorruptRecordColumnPresent(dataset)) {
+ throw new HoodieValidationException(String.format("Invalid condition, columnName=%s "
+ + "is not present in transformer " + "output schema", ERROR_TABLE_CURRUPT_RECORD_COL_NAME));
+ }
+ }
+
+ public static Dataset<Row> addNullValueErrorTableCorruptRecordColumn(Dataset<Row> dataset) {
+ if (!isErrorTableCorruptRecordColumnPresent(dataset)) {
+ dataset = dataset.withColumn(ERROR_TABLE_CURRUPT_RECORD_COL_NAME, lit(null));
+ }
+ return dataset;
+ }
+
+ private static boolean isErrorTableCorruptRecordColumnPresent(Dataset<Row> dataset) {
+ return Arrays.stream(dataset.columns()).collect(Collectors.toList())
Review Comment:
can we simplify this
```
df.columns.contains(ERROR_TABLE_CURRUPT_RECORD_COL_NAME)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1185684016
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -28,32 +28,39 @@
import org.apache.hudi.exception.HoodieException;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.exception.HoodieValidationException;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
+import static org.apache.spark.sql.functions.lit;
+
import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;
+import java.util.stream.Collectors;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_CLASS;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_FAILURE_STRATEGY;
+import static org.apache.hudi.utilities.deltastreamer.BaseErrorTableWriter.ERROR_TABLE_CURRUPT_RECORD_COL_NAME;
Review Comment:
`_corrupt_record ` this column is intermediate state , it's never commited to base table as column . before commiting error table , we separate out valid / error events and drop this column .
https://issues.apache.org/jira/browse/HUDI-6181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1517315529
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a7c71742f4b8fdc416b885edd0bb3e63687a715 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1175977700
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/ChainedErrorTableTransformer.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.transform;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.utilities.deltastreamer.ErrorTableUtils;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.List;
+
+/**
+ * A {@link Transformer} to chain other {@link Transformer}s and apply sequentially.
+ * Adds errorTableCorruptRecordColumn at the beginning of transformations and validates
+ * if that column is not dropped in any of the transformations.
+ */
+public class ChainedErrorTableTransformer extends ChainedTransformer {
Review Comment:
i think `ErrorTableAwareChainedTransformer` sounds more accurate, as it's not a transformer intended for error table itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1549511667
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3cd1d7cad018c67c90ba539d48813d94ac1d1c35 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788)
* 4ace27d8182562b768d28ccfe34dfc9611706b7d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114)
* f540c4e6eac618acfa15c5ff03d53beac3461341 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1524122758
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 56550641906350ac03288ad2f23fff15ade2e9da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1523354987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aea97f3976876ed811329b4e7892b99ae97e0bab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669)
* 56550641906350ac03288ad2f23fff15ade2e9da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] harsh1231 commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "harsh1231 (via GitHub)" <gi...@apache.org>.
harsh1231 commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1182932545
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -28,32 +28,39 @@
import org.apache.hudi.exception.HoodieException;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.exception.HoodieValidationException;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
+import static org.apache.spark.sql.functions.lit;
+
import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;
+import java.util.stream.Collectors;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_CLASS;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_FAILURE_STRATEGY;
+import static org.apache.hudi.utilities.deltastreamer.BaseErrorTableWriter.ERROR_TABLE_CURRUPT_RECORD_COL_NAME;
Review Comment:
`ERROR_TABLE_CURRUPT_RECORD_COL_NAME` currently this is fixed column . which is same as spark default `_corrupt_record` . We should make this configurable .
` incase of incremental consumer` -> only if it is part of incoming dataset .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8520:
URL: https://github.com/apache/hudi/pull/8520#discussion_r1182950555
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/ErrorTableUtils.java:
##########
@@ -28,32 +28,39 @@
import org.apache.hudi.exception.HoodieException;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.exception.HoodieValidationException;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
+import static org.apache.spark.sql.functions.lit;
+
import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;
+import java.util.stream.Collectors;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_CLASS;
import static org.apache.hudi.config.HoodieErrorTableConfig.ERROR_TABLE_WRITE_FAILURE_STRATEGY;
+import static org.apache.hudi.utilities.deltastreamer.BaseErrorTableWriter.ERROR_TABLE_CURRUPT_RECORD_COL_NAME;
Review Comment:
can we file a follow up task for making it configurable.
btw, I don't understand your response on incremental consumer question. can u help clarify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1551163079
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
}, {
"hash" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143",
"triggerID" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8520:
URL: https://github.com/apache/hudi/pull/8520#issuecomment-1553567992
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16508",
"triggerID" : "f42d020cc32a23c69e72f6be0f3f5b92d54f0b99",
"triggerType" : "PUSH"
}, {
"hash" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16509",
"triggerID" : "6a7c71742f4b8fdc416b885edd0bb3e63687a715",
"triggerType" : "PUSH"
}, {
"hash" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16608",
"triggerID" : "5ca6c38df3f6511f072a1c65ac291249f0957311",
"triggerType" : "PUSH"
}, {
"hash" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16669",
"triggerID" : "aea97f3976876ed811329b4e7892b99ae97e0bab",
"triggerType" : "PUSH"
}, {
"hash" : "56550641906350ac03288ad2f23fff15ade2e9da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16684",
"triggerID" : "56550641906350ac03288ad2f23fff15ade2e9da",
"triggerType" : "PUSH"
}, {
"hash" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16788",
"triggerID" : "3cd1d7cad018c67c90ba539d48813d94ac1d1c35",
"triggerType" : "PUSH"
}, {
"hash" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17114",
"triggerID" : "4ace27d8182562b768d28ccfe34dfc9611706b7d",
"triggerType" : "PUSH"
}, {
"hash" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17123",
"triggerID" : "f540c4e6eac618acfa15c5ff03d53beac3461341",
"triggerType" : "PUSH"
}, {
"hash" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17143",
"triggerID" : "79139a3aeb690e1d62cbbeb2da79f5fcc9a8b244",
"triggerType" : "PUSH"
}, {
"hash" : "0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17185",
"triggerID" : "0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ad850ba8e43e954d5a83ffb6a4e68bc3e3dd68b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #8520: [HUDI-6115] Hardening expectation of corruptRecordColumn in ChainedTransformer.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #8520:
URL: https://github.com/apache/hudi/pull/8520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org