You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "geserdugarov (via GitHub)" <gi...@apache.org> on 2024/03/04 11:53:07 UTC
[PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
geserdugarov opened a new pull request, #10810:
URL: https://github.com/apache/hudi/pull/10810
### Change Logs
Current implementation of `HoodieSchemaUtils.deduceWriterSchema` is really complicated with many flags used in one place.
To figure out how this method is working, we need to draw directed acyclic graph. Here the current implementation:

Yellow is used for marking flags, and purple is used for marking returned schemas or exceptions.
We couldn't remove any of the used flags without changing processing logic. The only possibility is to refactor `deduceWriterSchema()` for simplification, and preparing for future removing of deprecated `hoodie.datasource.write.reconcile.schema`.
This MR proposes to consider the following changed processing schema:

### Impact
Refactored `HoodieSchemaUtils.deduceWriterSchema`.
### Risk level (write none, low medium or high below)
Low.
### Documentation Update
No need.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1980838047
Please, look at diffs in a split regime. The main refactoring is in the `deduceWriterSchemaWithoutReconcile()` method, simplified to one if-else:
https://github.com/geserdugarov/hudi-open-source/blob/6e2278541811793cc208b4da940444103245da36/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSchemaUtils.scala#L116C1-L151C4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1980805895
@jonvex , hi!
After your last [commit](https://github.com/apache/hudi/commit/09979f67046c6a70e6501c5d8bc7e74d9f8e532b) I need to update this MR. Updated schema:

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1981199123
@geserdugarov thanks for the contribution! I will review this today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1980859367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
}, {
"hash" : "137b32afec91ddcf966402cba1038b65e69e089f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790",
"triggerID" : "137b32afec91ddcf966402cba1038b65e69e089f",
"triggerType" : "PUSH"
}, {
"hash" : "6e2278541811793cc208b4da940444103245da36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6e2278541811793cc208b4da940444103245da36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 137b32afec91ddcf966402cba1038b65e69e089f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790)
* 6e2278541811793cc208b4da940444103245da36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1976440075
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0f339942ebfc108d112bcb6b213565d1cb117bf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1980872895
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
}, {
"hash" : "137b32afec91ddcf966402cba1038b65e69e089f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790",
"triggerID" : "137b32afec91ddcf966402cba1038b65e69e089f",
"triggerType" : "PUSH"
}, {
"hash" : "6e2278541811793cc208b4da940444103245da36",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22815",
"triggerID" : "6e2278541811793cc208b4da940444103245da36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 137b32afec91ddcf966402cba1038b65e69e089f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790)
* 6e2278541811793cc208b4da940444103245da36 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #10810:
URL: https://github.com/apache/hudi/pull/10810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1976534646
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0f339942ebfc108d112bcb6b213565d1cb117bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1978291039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
}, {
"hash" : "137b32afec91ddcf966402cba1038b65e69e089f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "137b32afec91ddcf966402cba1038b65e69e089f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0f339942ebfc108d112bcb6b213565d1cb117bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774)
* 137b32afec91ddcf966402cba1038b65e69e089f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1978358908
Hmm, good code quality is what we always in persuit with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov closed pull request #10810: [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags
URL: https://github.com/apache/hudi/pull/10810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1976454430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0f339942ebfc108d112bcb6b213565d1cb117bf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1978307061
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
}, {
"hash" : "137b32afec91ddcf966402cba1038b65e69e089f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790",
"triggerID" : "137b32afec91ddcf966402cba1038b65e69e089f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0f339942ebfc108d112bcb6b213565d1cb117bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774)
* 137b32afec91ddcf966402cba1038b65e69e089f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1978512701
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
}, {
"hash" : "137b32afec91ddcf966402cba1038b65e69e089f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790",
"triggerID" : "137b32afec91ddcf966402cba1038b65e69e089f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 137b32afec91ddcf966402cba1038b65e69e089f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6947] Refactored HoodieSchemaUtils.deduceWriterSchema with many flags [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10810:
URL: https://github.com/apache/hudi/pull/10810#issuecomment-1981093368
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22774",
"triggerID" : "d0f339942ebfc108d112bcb6b213565d1cb117bf",
"triggerType" : "PUSH"
}, {
"hash" : "137b32afec91ddcf966402cba1038b65e69e089f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22790",
"triggerID" : "137b32afec91ddcf966402cba1038b65e69e089f",
"triggerType" : "PUSH"
}, {
"hash" : "6e2278541811793cc208b4da940444103245da36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22815",
"triggerID" : "6e2278541811793cc208b4da940444103245da36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6e2278541811793cc208b4da940444103245da36 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org