You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2024/12/17 23:00:08 UTC
[PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
linliu-code opened a new pull request, #12510:
URL: https://github.com/apache/hudi/pull/12510
### Change Logs
Add specific configs for schema evolution.
### Impact
Support schema evolution better.
### Risk level (write none, low medium or high below)
Low.
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12510:
URL: https://github.com/apache/hudi/pull/12510#discussion_r1891057223
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/SparkBroadcastManager.java:
##########
@@ -126,4 +146,24 @@ static Configuration getHadoopConfiguration(Configuration configuration) {
}
return (new HadoopStorageConfiguration(hadoopConf).getInline()).unwrap();
}
+
+ static Map<String, String> getSchemaEvolutionConfigs(TableSchemaResolver schemaResolver,
+ HoodieTimeline timeline,
+ InstantFileNameGenerator fileNameGenerator,
+ String basePath) {
+ Option<InternalSchema> internalSchemaOpt = schemaResolver.getTableInternalSchemaFromCommitMetadata();
+ Map<String, String> configs = new HashMap<>();
+ if (internalSchemaOpt.isPresent()) {
+ List<String> instantFiles = timeline.getInstants().stream().map(fileNameGenerator::getFileName).collect(Collectors.toList());
+ configs.put(SparkInternalSchemaConverter.HOODIE_VALID_COMMITS_LIST, String.join(",", instantFiles));
Review Comment:
Not sure why instant file name is needed, should we just just the instant timestamp string instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552226196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
}, {
"hash" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 944a155da0af2d0045e45ed26be3e6dd6e1fb033 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472)
* e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498)
* 63c5c2d7d407b49a566c64307a166d7f0f54dbf4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2549984711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbf531b1e80230419de628090fdf3e5578a3d3d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552169056
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 944a155da0af2d0045e45ed26be3e6dd6e1fb033 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472)
* e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2549837092
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbf531b1e80230419de628090fdf3e5578a3d3d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2550002938
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbf531b1e80230419de628090fdf3e5578a3d3d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471)
* 944a155da0af2d0045e45ed26be3e6dd6e1fb033 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2550121305
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 944a155da0af2d0045e45ed26be3e6dd6e1fb033 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2549839609
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbf531b1e80230419de628090fdf3e5578a3d3d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2550004755
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbf531b1e80230419de628090fdf3e5578a3d3d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471)
* 944a155da0af2d0045e45ed26be3e6dd6e1fb033 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552166434
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 944a155da0af2d0045e45ed26be3e6dd6e1fb033 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472)
* e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex merged PR #12510:
URL: https://github.com/apache/hudi/pull/12510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #12510:
URL: https://github.com/apache/hudi/pull/12510#discussion_r1892566860
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/SparkBroadcastManager.java:
##########
@@ -126,4 +146,24 @@ static Configuration getHadoopConfiguration(Configuration configuration) {
}
return (new HadoopStorageConfiguration(hadoopConf).getInline()).unwrap();
}
+
+ static Map<String, String> getSchemaEvolutionConfigs(TableSchemaResolver schemaResolver,
+ HoodieTimeline timeline,
+ InstantFileNameGenerator fileNameGenerator,
+ String basePath) {
+ Option<InternalSchema> internalSchemaOpt = schemaResolver.getTableInternalSchemaFromCommitMetadata();
+ Map<String, String> configs = new HashMap<>();
+ if (internalSchemaOpt.isPresent()) {
+ List<String> instantFiles = timeline.getInstants().stream().map(fileNameGenerator::getFileName).collect(Collectors.toList());
+ configs.put(SparkInternalSchemaConverter.HOODIE_VALID_COMMITS_LIST, String.join(",", instantFiles));
Review Comment:
https://github.com/apache/hudi/blob/9da3221a79465f3326ae3ac206b08d60864ddcaa/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala#L678 I'm sure I just copied this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #12510:
URL: https://github.com/apache/hudi/pull/12510#discussion_r1891097963
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/SparkBroadcastManager.java:
##########
@@ -126,4 +146,24 @@ static Configuration getHadoopConfiguration(Configuration configuration) {
}
return (new HadoopStorageConfiguration(hadoopConf).getInline()).unwrap();
}
+
+ static Map<String, String> getSchemaEvolutionConfigs(TableSchemaResolver schemaResolver,
+ HoodieTimeline timeline,
+ InstantFileNameGenerator fileNameGenerator,
+ String basePath) {
+ Option<InternalSchema> internalSchemaOpt = schemaResolver.getTableInternalSchemaFromCommitMetadata();
+ Map<String, String> configs = new HashMap<>();
+ if (internalSchemaOpt.isPresent()) {
+ List<String> instantFiles = timeline.getInstants().stream().map(fileNameGenerator::getFileName).collect(Collectors.toList());
+ configs.put(SparkInternalSchemaConverter.HOODIE_VALID_COMMITS_LIST, String.join(",", instantFiles));
Review Comment:
@jonvex should be able to answer since Jon added this for fg reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552264979
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
}, {
"hash" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499",
"triggerID" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498)
* 63c5c2d7d407b49a566c64307a166d7f0f54dbf4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552354988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
}, {
"hash" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499",
"triggerID" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"triggerType" : "PUSH"
}, {
"hash" : "1be400a19e5364d61a034aabdd9d584573bcb157",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2501",
"triggerID" : "1be400a19e5364d61a034aabdd9d584573bcb157",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63c5c2d7d407b49a566c64307a166d7f0f54dbf4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499)
* 1be400a19e5364d61a034aabdd9d584573bcb157 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2501)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552280834
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
}, {
"hash" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499",
"triggerID" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"triggerType" : "PUSH"
}, {
"hash" : "1be400a19e5364d61a034aabdd9d584573bcb157",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1be400a19e5364d61a034aabdd9d584573bcb157",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498)
* 63c5c2d7d407b49a566c64307a166d7f0f54dbf4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499)
* 1be400a19e5364d61a034aabdd9d584573bcb157 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552283264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
}, {
"hash" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499",
"triggerID" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"triggerType" : "PUSH"
}, {
"hash" : "1be400a19e5364d61a034aabdd9d584573bcb157",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1be400a19e5364d61a034aabdd9d584573bcb157",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63c5c2d7d407b49a566c64307a166d7f0f54dbf4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499)
* 1be400a19e5364d61a034aabdd9d584573bcb157 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552228535
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
}, {
"hash" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498)
* 63c5c2d7d407b49a566c64307a166d7f0f54dbf4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12510:
URL: https://github.com/apache/hudi/pull/12510#issuecomment-2552448083
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2471",
"triggerID" : "afbf531b1e80230419de628090fdf3e5578a3d3d",
"triggerType" : "PUSH"
}, {
"hash" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2472",
"triggerID" : "944a155da0af2d0045e45ed26be3e6dd6e1fb033",
"triggerType" : "PUSH"
}, {
"hash" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2498",
"triggerID" : "e14e51a6973eeeb6f591ac8a0fb21c0836aa0b9c",
"triggerType" : "PUSH"
}, {
"hash" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2499",
"triggerID" : "63c5c2d7d407b49a566c64307a166d7f0f54dbf4",
"triggerType" : "PUSH"
}, {
"hash" : "1be400a19e5364d61a034aabdd9d584573bcb157",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2501",
"triggerID" : "1be400a19e5364d61a034aabdd9d584573bcb157",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1be400a19e5364d61a034aabdd9d584573bcb157 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2501)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8776] Add schema evolution configs to SparkBroadcastManager [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12510:
URL: https://github.com/apache/hudi/pull/12510#discussion_r1891057223
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/SparkBroadcastManager.java:
##########
@@ -126,4 +146,24 @@ static Configuration getHadoopConfiguration(Configuration configuration) {
}
return (new HadoopStorageConfiguration(hadoopConf).getInline()).unwrap();
}
+
+ static Map<String, String> getSchemaEvolutionConfigs(TableSchemaResolver schemaResolver,
+ HoodieTimeline timeline,
+ InstantFileNameGenerator fileNameGenerator,
+ String basePath) {
+ Option<InternalSchema> internalSchemaOpt = schemaResolver.getTableInternalSchemaFromCommitMetadata();
+ Map<String, String> configs = new HashMap<>();
+ if (internalSchemaOpt.isPresent()) {
+ List<String> instantFiles = timeline.getInstants().stream().map(fileNameGenerator::getFileName).collect(Collectors.toList());
+ configs.put(SparkInternalSchemaConverter.HOODIE_VALID_COMMITS_LIST, String.join(",", instantFiles));
Review Comment:
Not sure why instant file name is needed, should we just use the instant timestamp string instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org