You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/22 21:21:50 UTC
[GitHub] [hudi] jonvex opened a new pull request, #6753: Deltastreamer table config change validation
jonvex opened a new pull request, #6753:
URL: https://github.com/apache/hudi/pull/6753
### Change Logs
Table configuration properties were not being validated in deltastreamer so changes to properties that shouldn't change were being allowed. To fix this, we now call the validator that is used by spark sql writer.
### Impact
Now, a user should not be able to change immutable configs for a table which will silently cause issues down the line.
**Risk level: low **
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1256630461
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36991b80eba4e400ecac7ecdda6d16d757d71dbb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591)
* 6ef673e4b7b41c5b1a046065a2b512354b4f1553 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1258731786
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d151481a5f9caec1f09f87a0f71e66c4536562b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673)
* 5095da393ba4fbc3cf056f8a472e6fd19de2eb2b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1261429040
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
}, {
"hash" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826",
"triggerID" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"triggerType" : "PUSH"
}, {
"hash" : "722ee61ed665a6e31cc4871bfd71c17f22673868",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11846",
"triggerID" : "722ee61ed665a6e31cc4871bfd71c17f22673868",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 722ee61ed665a6e31cc4871bfd71c17f22673868 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11846)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260408466
I triaged as to why payloadClassUpdate test in deltastreamer was succeeding w/ this patch. I was expecting it to fail. apparently payload class is a top level config in deltastreamer and hence its not passed via regular props. So, it bypasses the validation we have. for now, lets proceed w/ the patch. later we can decide if we really need to add validation for payload class update as well.
also, have reverted m1 pom changes. once CI is green, we are good to land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1258871786
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5095da393ba4fbc3cf056f8a472e6fd19de2eb2b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6753:
URL: https://github.com/apache/hudi/pull/6753#discussion_r978224183
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java:
##########
@@ -435,92 +440,92 @@ public boolean equals(Object o) {
}
Config config = (Config) o;
return sourceLimit == config.sourceLimit
- && Objects.equals(targetBasePath, config.targetBasePath)
- && Objects.equals(targetTableName, config.targetTableName)
- && Objects.equals(tableType, config.tableType)
- && Objects.equals(baseFileFormat, config.baseFileFormat)
- && Objects.equals(propsFilePath, config.propsFilePath)
- && Objects.equals(configs, config.configs)
- && Objects.equals(sourceClassName, config.sourceClassName)
- && Objects.equals(sourceOrderingField, config.sourceOrderingField)
- && Objects.equals(payloadClassName, config.payloadClassName)
- && Objects.equals(schemaProviderClassName, config.schemaProviderClassName)
- && Objects.equals(transformerClassNames, config.transformerClassNames)
- && operation == config.operation
- && Objects.equals(filterDupes, config.filterDupes)
- && Objects.equals(enableHiveSync, config.enableHiveSync)
- && Objects.equals(enableMetaSync, config.enableMetaSync)
- && Objects.equals(forceEmptyMetaSync, config.forceEmptyMetaSync)
- && Objects.equals(syncClientToolClassNames, config.syncClientToolClassNames)
- && Objects.equals(maxPendingCompactions, config.maxPendingCompactions)
- && Objects.equals(maxPendingClustering, config.maxPendingClustering)
- && Objects.equals(continuousMode, config.continuousMode)
- && Objects.equals(minSyncIntervalSeconds, config.minSyncIntervalSeconds)
- && Objects.equals(sparkMaster, config.sparkMaster)
- && Objects.equals(commitOnErrors, config.commitOnErrors)
- && Objects.equals(deltaSyncSchedulingWeight, config.deltaSyncSchedulingWeight)
- && Objects.equals(compactSchedulingWeight, config.compactSchedulingWeight)
- && Objects.equals(clusterSchedulingWeight, config.clusterSchedulingWeight)
- && Objects.equals(deltaSyncSchedulingMinShare, config.deltaSyncSchedulingMinShare)
- && Objects.equals(compactSchedulingMinShare, config.compactSchedulingMinShare)
- && Objects.equals(clusterSchedulingMinShare, config.clusterSchedulingMinShare)
- && Objects.equals(forceDisableCompaction, config.forceDisableCompaction)
- && Objects.equals(checkpoint, config.checkpoint)
- && Objects.equals(initialCheckpointProvider, config.initialCheckpointProvider)
- && Objects.equals(help, config.help);
+ && Objects.equals(targetBasePath, config.targetBasePath)
Review Comment:
can you revert unintended changes.
##########
packaging/hudi-hive-sync-bundle/pom.xml:
##########
@@ -72,6 +72,7 @@
<include>org.apache.hudi:hudi-hive-sync</include>
<include>com.beust:jcommander</include>
+ <include>org.apache.avro:avro</include>
Review Comment:
lets revert this in this PR. we have a separate PR tackling this
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -1447,8 +1492,8 @@ public void testPayloadClassUpdate() throws Exception {
public void testPartialPayloadClass() throws Exception {
String dataSetBasePath = dfsBasePath + "/test_dataset_mor";
HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(dataSetBasePath, WriteOperationType.BULK_INSERT,
- Collections.singletonList(SqlQueryBasedTransformer.class.getName()), PROPS_FILENAME_TEST_SOURCE, false,
- true, true, PartialUpdateAvroPayload.class.getName(), "MERGE_ON_READ");
+ Collections.singletonList(SqlQueryBasedTransformer.class.getName()), PROPS_FILENAME_TEST_SOURCE, false,
Review Comment:
again, please do revert unintended changes.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java:
##########
@@ -651,6 +656,9 @@ public DeltaSyncService(Config cfg, JavaSparkContext jssc, FileSystem fs, Config
+ cfg.baseFileFormat);
cfg.baseFileFormat = baseFileFormat;
this.cfg.baseFileFormat = baseFileFormat;
+ HashMap<String,String> scalaprops = new HashMap<>();
Review Comment:
we can probably name this "props" or "properties". its actually scala map only in L661.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -664,6 +664,51 @@ public void testBulkInsertsAndUpsertsWithBootstrap() throws Exception {
assertTrue(fieldNames.containsAll(expectedFieldNames));
}
+ @Test
+ public void testModifiedTableConfigs() throws Exception {
+ String tableBasePath = dfsBasePath + "/test_table";
+
+ // Initial bulk insert
+ HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath, WriteOperationType.BULK_INSERT);
+ new HoodieDeltaStreamer(cfg, jsc).sync();
+ TestHelpers.assertRecordCount(1000, tableBasePath, sqlContext);
+ TestHelpers.assertDistanceCount(1000, tableBasePath, sqlContext);
+ TestHelpers.assertCommitMetadata("00000", tableBasePath, dfs, 1);
+
+ // No new data => no commits.
+ cfg.sourceLimit = 0;
+ new HoodieDeltaStreamer(cfg, jsc).sync();
+ TestHelpers.assertRecordCount(1000, tableBasePath, sqlContext);
+ TestHelpers.assertDistanceCount(1000, tableBasePath, sqlContext);
+ TestHelpers.assertCommitMetadata("00000", tableBasePath, dfs, 1);
+
+ // add disallowed config update to recordkey field. An exception should be thrown
+ cfg.sourceLimit = 2000;
+ cfg.operation = WriteOperationType.UPSERT;
+ cfg.configs.add(HoodieTableConfig.RECORDKEY_FIELDS.key() + "=differentval");
+ assertThrows(HoodieException.class, () -> new HoodieDeltaStreamer(cfg, jsc).sync());
+
+ TestHelpers.assertRecordCount(1000, tableBasePath, sqlContext);
Review Comment:
we can probably write the test differently and reduce the no of lines. lets sync up sometime and go over this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260416235
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
}, {
"hash" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826",
"triggerID" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805)
* 9dd151db04f82d1ffa88e0fb32ce47424a5fb485 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260323649
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1261094538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
}, {
"hash" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826",
"triggerID" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"triggerType" : "PUSH"
}, {
"hash" : "722ee61ed665a6e31cc4871bfd71c17f22673868",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "722ee61ed665a6e31cc4871bfd71c17f22673868",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9dd151db04f82d1ffa88e0fb32ce47424a5fb485 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826)
* 722ee61ed665a6e31cc4871bfd71c17f22673868 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1261101062
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
}, {
"hash" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826",
"triggerID" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"triggerType" : "PUSH"
}, {
"hash" : "722ee61ed665a6e31cc4871bfd71c17f22673868",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11846",
"triggerID" : "722ee61ed665a6e31cc4871bfd71c17f22673868",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9dd151db04f82d1ffa88e0fb32ce47424a5fb485 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826)
* 722ee61ed665a6e31cc4871bfd71c17f22673868 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11846)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260413049
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
}, {
"hash" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805)
* 9dd151db04f82d1ffa88e0fb32ce47424a5fb485 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1256634714
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36991b80eba4e400ecac7ecdda6d16d757d71dbb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591)
* 6ef673e4b7b41c5b1a046065a2b512354b4f1553 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1258716198
have pushed out a commit to address minor comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260047809
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 364470c22551fb8fc17c4d7286e6b4504198d614 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800)
* f3bd19a7f74e4612058cd1eec09c26893d810e25 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
codope commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260389144
simply rebased as the flaky test was fixed recently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1261469914
azure CI failed due to flaky flink IT failure. going ahead w/ merge as we have code freeze coming up. changes in the patch is safe to land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1256749767
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6ef673e4b7b41c5b1a046065a2b512354b4f1553 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1256834882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d151481a5f9caec1f09f87a0f71e66c4536562b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1256774552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6ef673e4b7b41c5b1a046065a2b512354b4f1553 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654)
* 7d151481a5f9caec1f09f87a0f71e66c4536562b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1259965228
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5095da393ba4fbc3cf056f8a472e6fd19de2eb2b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753)
* 364470c22551fb8fc17c4d7286e6b4504198d614 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1255599048
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36991b80eba4e400ecac7ecdda6d16d757d71dbb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1255646947
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36991b80eba4e400ecac7ecdda6d16d757d71dbb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6753:
URL: https://github.com/apache/hudi/pull/6753#discussion_r980554115
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -664,6 +655,43 @@ public void testBulkInsertsAndUpsertsWithBootstrap() throws Exception {
assertTrue(fieldNames.containsAll(expectedFieldNames));
}
+ @Test
+ public void testModifiedTableConfigs() throws Exception {
+ String tableBasePath = dfsBasePath + "/test_table";
+
+ // Initial bulk insert
+ HoodieDeltaStreamer.Config cfg = TestHelpers.makeConfig(tableBasePath, WriteOperationType.BULK_INSERT);
+ testRecordCount(cfg,1000,tableBasePath,"00000",1);
+
+ // No new data => no commits.
+ cfg.sourceLimit = 0;
+ testRecordCount(cfg,1000,tableBasePath,"00000",1);
+
+ // add disallowed config update to recordkey field. An exception should be thrown
+ cfg.sourceLimit = 2000;
+ cfg.operation = WriteOperationType.UPSERT;
+ cfg.configs.add(HoodieTableConfig.RECORDKEY_FIELDS.key() + "=differentval");
+ assertThrows(HoodieException.class, () -> testRecordCount(cfg,1000,tableBasePath,"00000",1));
+ List<Row> counts = TestHelpers.countsPerCommit(tableBasePath, sqlContext);
+ assertEquals(1000, counts.stream().mapToLong(entry -> entry.getLong(1)).sum());
+
+
+ //perform the upsert and now with the original config, the commit should go through
+ HoodieDeltaStreamer.Config newCfg = TestHelpers.makeConfig(tableBasePath, WriteOperationType.BULK_INSERT);
+ newCfg.sourceLimit = 2000;
+ newCfg.operation = WriteOperationType.UPSERT;
+ testRecordCount(newCfg,1950,tableBasePath,"00001",2);
+ List<Row> counts2 = TestHelpers.countsPerCommit(tableBasePath, sqlContext);
+ assertEquals(1950, counts2.stream().mapToLong(entry -> entry.getLong(1)).sum());
+ }
+
+ private void testRecordCount(HoodieDeltaStreamer.Config cfg, Integer expected, String tableBasePath, String metadata, Integer totalCommits) throws Exception {
Review Comment:
a good name would be "syncAndAssertRecordCount"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260042714
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5095da393ba4fbc3cf056f8a472e6fd19de2eb2b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753)
* 364470c22551fb8fc17c4d7286e6b4504198d614 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800)
* f3bd19a7f74e4612058cd1eec09c26893d810e25 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260109296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3bd19a7f74e4612058cd1eec09c26893d810e25 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802)
* bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1256766864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6ef673e4b7b41c5b1a046065a2b512354b4f1553 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654)
* 7d151481a5f9caec1f09f87a0f71e66c4536562b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1258735127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d151481a5f9caec1f09f87a0f71e66c4536562b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673)
* 5095da393ba4fbc3cf056f8a472e6fd19de2eb2b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1255594682
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36991b80eba4e400ecac7ecdda6d16d757d71dbb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260105388
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 364470c22551fb8fc17c4d7286e6b4504198d614 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800)
* f3bd19a7f74e4612058cd1eec09c26893d810e25 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802)
* bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1259970552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5095da393ba4fbc3cf056f8a472e6fd19de2eb2b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753)
* 364470c22551fb8fc17c4d7286e6b4504198d614 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6753:
URL: https://github.com/apache/hudi/pull/6753#discussion_r982256620
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java:
##########
@@ -651,6 +656,9 @@ public DeltaSyncService(Config cfg, JavaSparkContext jssc, FileSystem fs, Config
+ cfg.baseFileFormat);
cfg.baseFileFormat = baseFileFormat;
this.cfg.baseFileFormat = baseFileFormat;
+ HashMap<String,String> propsToValidate = new HashMap<>();
Review Comment:
should declare Map instead of concrete impl. class.
##########
pom.xml:
##########
@@ -145,6 +145,7 @@
<flink.clients.artifactId>flink-clients</flink.clients.artifactId>
<flink.connector.kafka.artifactId>flink-connector-kafka</flink.connector.kafka.artifactId>
<flink.hadoop.compatibility.artifactId>flink-hadoop-compatibility_2.12</flink.hadoop.compatibility.artifactId>
+ <rocksdbjni.version>5.17.2</rocksdbjni.version>
Review Comment:
although no risk, we should avoid having irrelevant change mixed in a patch that meant for other issue.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java:
##########
@@ -651,6 +656,9 @@ public DeltaSyncService(Config cfg, JavaSparkContext jssc, FileSystem fs, Config
+ cfg.baseFileFormat);
cfg.baseFileFormat = baseFileFormat;
this.cfg.baseFileFormat = baseFileFormat;
+ HashMap<String,String> propsToValidate = new HashMap<>();
+ properties.get().forEach((k,v) -> propsToValidate.put(k.toString(),v.toString()));
+ HoodieWriterUtils.validateTableConfig(this.sparkSession, JavaConverters.mapAsScalaMapConverter(propsToValidate).asScala().toMap(Predef.conforms()), meta.getTableConfig());
Review Comment:
could have used `org.apache.hudi.HoodieConversionUtils#mapAsScalaImmutableMap` to reduce code verbosity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6753:
URL: https://github.com/apache/hudi/pull/6753#issuecomment-1260870327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11591",
"triggerID" : "36991b80eba4e400ecac7ecdda6d16d757d71dbb",
"triggerType" : "PUSH"
}, {
"hash" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11654",
"triggerID" : "6ef673e4b7b41c5b1a046065a2b512354b4f1553",
"triggerType" : "PUSH"
}, {
"hash" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11673",
"triggerID" : "7d151481a5f9caec1f09f87a0f71e66c4536562b",
"triggerType" : "PUSH"
}, {
"hash" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11753",
"triggerID" : "5095da393ba4fbc3cf056f8a472e6fd19de2eb2b",
"triggerType" : "PUSH"
}, {
"hash" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11800",
"triggerID" : "364470c22551fb8fc17c4d7286e6b4504198d614",
"triggerType" : "PUSH"
}, {
"hash" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11802",
"triggerID" : "f3bd19a7f74e4612058cd1eec09c26893d810e25",
"triggerType" : "PUSH"
}, {
"hash" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11805",
"triggerID" : "bd1d6a2403336f0863b4e4ccb4f7a46eec76a4e3",
"triggerType" : "PUSH"
}, {
"hash" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826",
"triggerID" : "9dd151db04f82d1ffa88e0fb32ce47424a5fb485",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9dd151db04f82d1ffa88e0fb32ce47424a5fb485 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11826)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #6753: [HUDI-4734] Deltastreamer table config change validation
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #6753:
URL: https://github.com/apache/hudi/pull/6753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org