You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/21 02:45:04 UTC
[GitHub] [hudi] yihua opened a new pull request, #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
yihua opened a new pull request, #5379:
URL: https://github.com/apache/hudi/pull/5379
## What is the purpose of the pull request
This PR fixes the projection logic around a nested field which is used as the pre-combined key field. The fix is to include the root level field, i.e., "a", for a nested field "a.b.c" that is required.
## Brief change log
- Changes the logic to include the root level field for a required nested field in `HoodieSparkUtils.getRequiredSchema`
- Adds tests to guard around the behavior for a nested field used as pre-combined key.
## Verify this pull request
This change added tests in `TestHoodieAvroUtils` and `TestMORDataSourceStorage`. `TestMORDataSourceStorage` contains tests that use nested field "fare.currency" as the pre-combined key. Before this change, the tests with nest fields fail. After this PR, the tests pass.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105120261
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
}, {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188)
* d24bcd682911017e9a08c5b385337dded5ad4719 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105797145
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5379:
URL: https://github.com/apache/hudi/pull/5379#discussion_r854734744
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala:
##########
@@ -322,7 +323,9 @@ object HoodieSparkUtils extends SparkAdapterSupport {
val name2Fields = tableAvroSchema.getFields.asScala.map(f => f.name() -> f).toMap
// Here have to create a new Schema.Field object
// to prevent throwing exceptions like "org.apache.avro.AvroRuntimeException: Field already used".
- val requiredFields = requiredColumns.map(c => name2Fields(c))
+ // For a nested field, we include the root-level field
+ val requiredFields = requiredColumns.map(c => HoodieAvroUtils.getRootLevelFieldName(c))
+ .distinct.map(c => name2Fields(c))
Review Comment:
this code path affects both MOR and COW, right? is it for spark later to drill down to specific nested cols?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5379:
URL: https://github.com/apache/hudi/pull/5379#discussion_r854743373
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala:
##########
@@ -322,7 +323,9 @@ object HoodieSparkUtils extends SparkAdapterSupport {
val name2Fields = tableAvroSchema.getFields.asScala.map(f => f.name() -> f).toMap
// Here have to create a new Schema.Field object
// to prevent throwing exceptions like "org.apache.avro.AvroRuntimeException: Field already used".
- val requiredFields = requiredColumns.map(c => name2Fields(c))
+ // For a nested field, we include the root-level field
+ val requiredFields = requiredColumns.map(c => HoodieAvroUtils.getRootLevelFieldName(c))
+ .distinct.map(c => name2Fields(c))
Review Comment:
we discussed the `distinct` looks problematic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1104888042
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105152335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
}, {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d24bcd682911017e9a08c5b385337dded5ad4719 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1104689728
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8184",
"triggerID" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"triggerType" : "PUSH"
}, {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 85985e330971e0dfdea39a48d5a6e862e86e2c6b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8184)
* dd8db55be78b9e056d658604404137bef48ba4fb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1104883300
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105052299
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5379:
URL: https://github.com/apache/hudi/pull/5379#discussion_r854764950
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -336,7 +335,11 @@ abstract class HoodieBaseRelation(val sqlContext: SQLContext,
}
protected final def appendMandatoryColumns(requestedColumns: Array[String]): Array[String] = {
- val missing = mandatoryColumns.filter(col => !requestedColumns.contains(col))
+ // For a nested field in mandatory columns, we should first get the root-level field, and then
+ // check for any missing column, as the requestedColumns should only contain root-level fields
+ // We should only append root-level field as well
+ val missing = mandatoryColumns.map(col => HoodieAvroUtils.getRootLevelFieldName(col))
Review Comment:
Let's do this filtering when we assign it (name `mandatoryColumns` is misleading otherwise)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105049263
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105867957
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
}, {
"hash" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8208",
"triggerID" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"triggerType" : "PUSH"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211",
"triggerID" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"triggerType" : "PUSH"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8214",
"triggerID" : "1105797145",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5aae2f9d2e69727ca24605ecea2e3c74d04d3f00 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8214)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #5379:
URL: https://github.com/apache/hudi/pull/5379#discussion_r854743124
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala:
##########
@@ -322,7 +323,9 @@ object HoodieSparkUtils extends SparkAdapterSupport {
val name2Fields = tableAvroSchema.getFields.asScala.map(f => f.name() -> f).toMap
// Here have to create a new Schema.Field object
// to prevent throwing exceptions like "org.apache.avro.AvroRuntimeException: Field already used".
- val requiredFields = requiredColumns.map(c => name2Fields(c))
+ // For a nested field, we include the root-level field
+ val requiredFields = requiredColumns.map(c => HoodieAvroUtils.getRootLevelFieldName(c))
+ .distinct.map(c => name2Fields(c))
Review Comment:
just MOR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1104652278
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8184",
"triggerID" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 85985e330971e0dfdea39a48d5a6e862e86e2c6b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105542149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d24bcd682911017e9a08c5b385337dded5ad4719 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105706604
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
}, {
"hash" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8208",
"triggerID" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d24bcd682911017e9a08c5b385337dded5ad4719 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199)
* 8d5d57656d974002cde457a7ce56d0be607dfd49 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1104952748
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1104688133
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8184",
"triggerID" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"triggerType" : "PUSH"
}, {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 85985e330971e0dfdea39a48d5a6e862e86e2c6b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8184)
* dd8db55be78b9e056d658604404137bef48ba4fb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105549453
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d24bcd682911017e9a08c5b385337dded5ad4719 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105090559
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #5379:
URL: https://github.com/apache/hudi/pull/5379#discussion_r854863511
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -336,7 +335,11 @@ abstract class HoodieBaseRelation(val sqlContext: SQLContext,
}
protected final def appendMandatoryColumns(requestedColumns: Array[String]): Array[String] = {
- val missing = mandatoryColumns.filter(col => !requestedColumns.contains(col))
+ // For a nested field in mandatory columns, we should first get the root-level field, and then
+ // check for any missing column, as the requestedColumns should only contain root-level fields
+ // We should only append root-level field as well
+ val missing = mandatoryColumns.map(col => HoodieAvroUtils.getRootLevelFieldName(col))
Review Comment:
If I understand it correctly, do you mean to say we can do the filtering of `mandatoryColumns` upon initialization of the class? That's not possible since we need to do on-the-fly filtering based on the passed-in `requestedColumns` which may vary when `buildScan` is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105751877
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
}, {
"hash" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8208",
"triggerID" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"triggerType" : "PUSH"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d5d57656d974002cde457a7ce56d0be607dfd49 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8208)
* Unknown: [CANCELED](TBD)
* 5aae2f9d2e69727ca24605ecea2e3c74d04d3f00 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105797898
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
}, {
"hash" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8208",
"triggerID" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"triggerType" : "PUSH"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211",
"triggerID" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"triggerType" : "PUSH"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8214",
"triggerID" : "1105797145",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5aae2f9d2e69727ca24605ecea2e3c74d04d3f00 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8214)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105760787
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
}, {
"hash" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8208",
"triggerID" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211",
"triggerID" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 5aae2f9d2e69727ca24605ecea2e3c74d04d3f00 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8211)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #5379:
URL: https://github.com/apache/hudi/pull/5379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1104650821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 85985e330971e0dfdea39a48d5a6e862e86e2c6b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105749537
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1104781873
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8184",
"triggerID" : "85985e330971e0dfdea39a48d5a6e862e86e2c6b",
"triggerType" : "PUSH"
}, {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105702795
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
}, {
"hash" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d24bcd682911017e9a08c5b385337dded5ad4719 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199)
* 8d5d57656d974002cde457a7ce56d0be607dfd49 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105117289
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188",
"triggerID" : "dd8db55be78b9e056d658604404137bef48ba4fb",
"triggerType" : "PUSH"
}, {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd8db55be78b9e056d658604404137bef48ba4fb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8188)
* d24bcd682911017e9a08c5b385337dded5ad4719 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105574107
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d24bcd682911017e9a08c5b385337dded5ad4719 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5379:
URL: https://github.com/apache/hudi/pull/5379#discussion_r854951818
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -336,7 +335,11 @@ abstract class HoodieBaseRelation(val sqlContext: SQLContext,
}
protected final def appendMandatoryColumns(requestedColumns: Array[String]): Array[String] = {
- val missing = mandatoryColumns.filter(col => !requestedColumns.contains(col))
+ // For a nested field in mandatory columns, we should first get the root-level field, and then
+ // check for any missing column, as the requestedColumns should only contain root-level fields
+ // We should only append root-level field as well
+ val missing = mandatoryColumns.map(col => HoodieAvroUtils.getRootLevelFieldName(col))
Review Comment:
i think Alexey meant a readability improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5379: [HUDI-3936] Fix projection for a nested field as pre-combined key
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5379:
URL: https://github.com/apache/hudi/pull/5379#issuecomment-1105754792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8199",
"triggerID" : "d24bcd682911017e9a08c5b385337dded5ad4719",
"triggerType" : "PUSH"
}, {
"hash" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8208",
"triggerID" : "8d5d57656d974002cde457a7ce56d0be607dfd49",
"triggerType" : "PUSH"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1105749537",
"triggerType" : "MANUAL"
}, {
"hash" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5aae2f9d2e69727ca24605ecea2e3c74d04d3f00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5aae2f9d2e69727ca24605ecea2e3c74d04d3f00 UNKNOWN
* Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org