You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "VitoMakarevich (via GitHub)" <gi...@apache.org> on 2024/06/13 18:45:56 UTC
[PR] Fix Hudi being able to read 2-level structure if explicit flag for wr… [hudi]
VitoMakarevich opened a new pull request, #11450:
URL: https://github.com/apache/hudi/pull/11450
If I have `"spark.hadoop.parquet.avro.write-old-list-structure", "false"` explicitly set - to being able to write nulls inside arrays(the only way), Hudi starts to write Parquets with the following schema inside:
```
required group internal_list (LIST) {
repeated group list {
required int64 element;
}
}
```
But if I had some files produced before setting `"spark.hadoop.parquet.avro.write-old-list-structure", "false"`, they have the following schema inside
```
required group internal_list (LIST) {
repeated int64 array;
}
```
And Hudi 0.14.x at least fails to read records from such file - failing with exception
`Caused by: java.lang.RuntimeException: Null-value for required field: `
Even though the contents of arrays is `not null`(it cannot be null in fact since Avro requires `spark.hadoop.parquet.avro.write-old-list-structure` = `false` to write `null`s.
### Expected behavior
Taken from Hudi 0.12.1(not sure what exactly broke that):
1. If I have a file with 2 level structure and update(not matter having nulls inside array or not - both produce the same) arrives with "spark.hadoop.parquet.avro.write-old-list-structure", "false" - overwrite it into 3 level.(**fails in 0.14.1**)
2. If I have 3 level structure with nulls and update cames(not matter with nulls or without) - read and write correctly
The simple reproduction of issue can be found here:
https://github.com/VitoMakarevich/hudi-issue-014
Highly likely, the problem appeared after Hudi made some changes, so values from Hadoop conf started to propagate into Reader instance(likely they were not propagated before).
### Change Logs
Added explicit override of `spark.hadoop.parquet.avro.write-old-list-structure` = `true` if file being read is old(has 2 level structure).
### Impact
Running tests to ensure no unexpected issues propagating.
### Risk level (write none, low medium or high below)
medium
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166879255
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2176194891
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
}, {
"hash" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445",
"triggerID" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"triggerType" : "PUSH"
}, {
"hash" : "c814458b48d8a33a2b5ebbb0355183e129be89f4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24449",
"triggerID" : "c814458b48d8a33a2b5ebbb0355183e129be89f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 0dbffbf92f2cb18861621be3e216e65a03129cf6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445)
* c814458b48d8a33a2b5ebbb0355183e129be89f4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2175605541
Also I see that this block
```
if (!legacyMode) {
requestedSchema = new MessageType(requestedSchema.getName(), convertLegacyMap(requestedSchema.getFields()));
}
```
is redundant - since in all the cases `requestedSchema` fetched after tuning AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE equals to schema coming from this block - meaning AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE does the correct conversion.
So this code block should be removed to not cause confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1640637334
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -62,6 +58,23 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
return new ReadContext(requestedSchema, readContext.getReadSupportMetadata());
}
+ /**
+ * Here we want set config with which file has been written.
+ * Even though user may have overwritten {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE},
+ * it's only applicable to how to produce new files(here is a read path).
+ * Later the config value {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE} will still be used
+ * to write new file according to the user preferences.
+ **/
+ private void adjustConfToReadWithFileProduceMode(Boolean isLegacyModeWrittenFile, Configuration configuration) {
+ if (isLegacyModeWrittenFile) {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
+ "true", "support reading avro from legacy map/list in parquet file");
+ } else {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
+ "false", "support reading avro from non-legacy map/list in parquet file");
+ }
Review Comment:
I personally think this is an OK fix. An alternative may be adding something like `HoodieAvroSchemaConverter` to rewrite schema within Hudi. But it would be too much for such a corner case and could become hard to maintain in the future when Parquet changes their `AvroSchemaConverter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2169038041
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 242cc9c34dbb23192c0c1e5d42381835c544b3e4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415)
* 469138465aa8110f8c1478914d09e286fdd9f2d7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168590073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* dbff5f176b79f5848598a3041540b2b965aafa2c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414)
* 242cc9c34dbb23192c0c1e5d42381835c544b3e4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2170533778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 469138465aa8110f8c1478914d09e286fdd9f2d7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417)
* a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168857089
> @VitoMakarevich could you also raise the same set of changes against [branch-0.x](https://github.com/apache/hudi/tree/branch-0.x), where future 0. x releases are based off? It would be good to run through the CI in `branch-0.x` too.
Working on this, had to rewrite some tests with other primitives available for the previous state,and will now run tests locally and push, would it make sense to make another PR into 0.14.1 directly?
I mean - we are now blocked from using 0.14.1, so this is my first priority, once it's done, I will be more calm to proceed with merging into the main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168010500
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec5dda2f96699fde05db8042ab8fc45fcbb88738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Hudi being able to read 2-level structure if explicit flag for wr… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166575387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Hudi being able to read 2-level structure if explicit flag for wr… [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166558699
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166784596
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2169069202
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 469138465aa8110f8c1478914d09e286fdd9f2d7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1639851878
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -46,11 +46,7 @@ public HoodieAvroReadSupport() {
@Override
public ReadContext init(Configuration configuration, Map<String, String> keyValueMetaData, MessageType fileSchema) {
boolean legacyMode = checkLegacyMode(fileSchema.getFields());
- // support non-legacy list
- if (!legacyMode && configuration.get(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE) == null) {
Review Comment:
This line has been extracted and now I'm forcefully applying configuration even if something is set by the user, since if the user has `AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE=true`, then the case with rewriting 3 levels into 2 levels will either fail/silent data loss - basically scenarios 5 and 6 from the test suite.
This is opposite use case as we faced initially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1638937534
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,10 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ if (legacyMode) {
Review Comment:
Yes, struggling to do this with all Hudi test rails, but slowly proceeding.
I'm not sure exactly understand what you mean since if I have this code block it will anyway pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166782435
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168214286
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec5dda2f96699fde05db8042ab8fc45fcbb88738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405)
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 2a19d81bf1116a341380ce7d7511fbdc1f64ba81 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406)
* 836c87e014c4f94657d5303d08c3e6ea5794fc72 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407)
* 91620c7b073209cc0cd7c70165fa2f4231112e44 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168230107
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 2a19d81bf1116a341380ce7d7511fbdc1f64ba81 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406)
* 836c87e014c4f94657d5303d08c3e6ea5794fc72 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407)
* 91620c7b073209cc0cd7c70165fa2f4231112e44 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408)
* efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1640398375
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,10 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ if (legacyMode) {
Review Comment:
Great. What I meant is that new tests should be added, which failed before this fix and succeed after this fix, which is what you're doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2175726334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
}, {
"hash" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445",
"triggerID" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 93ddf4151fedb0698e0c5c56d69b4b866626d393 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440)
* 0dbffbf92f2cb18861621be3e216e65a03129cf6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2174525812
@vinothchandar @xushiyan @nsivabalan can you take a look as you were approvers of [RFC-46](https://github.com/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2170755660
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168424198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 318d5e5f8d22e011b21ee14e770a5a9849334b7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1643313572
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -62,6 +58,23 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
return new ReadContext(requestedSchema, readContext.getReadSupportMetadata());
}
+ /**
+ * Here we want set config with which file has been written.
+ * Even though user may have overwritten {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE},
+ * it's only applicable to how to produce new files(here is a read path).
+ * Later the config value {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE} will still be used
+ * to write new file according to the user preferences.
+ **/
+ private void adjustConfToReadWithFileProduceMode(Boolean isLegacyModeWrittenFile, Configuration configuration) {
Review Comment:
nit: use `boolean` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166915644
In certain setups, this problem leads to silent data loss. https://github.com/VitoMakarevich/hudi-issue-014?tab=readme-ov-file#silent-dataloss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166930448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383)
* 01581df81e33432179d9dfe574f8e9ae74f18038 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1639440948
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,13 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ // If old file is written with legacy mode, read with legacy mode, even if now it's non-legacy mode in conf.
+ // Since by changing the property users want to control how files are written, not how they are read.
+ // Later the value of WRITE_OLD_LIST_STRUCTURE will be picked from conf, thus either keeping old mode or writing new mode.
+ if (legacyMode) {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
Review Comment:
This is what exactly leads to reading `null` from list fields even though they contain a value.
This leads to exception or silent data loss.
Basically - the change is for reading, while this setting is for writing. The writing part still will be used.
e.g. if I set WRITE_OLD_LIST_STRUCTURE to any value - I expect Hudi/Spark to be able to read 2 levels and 3 level while producing only 3 levels, but in the current state - Hudi reads nulls from 2 level parquets if the setting is set to false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2175829244
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
}, {
"hash" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445",
"triggerID" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 0dbffbf92f2cb18861621be3e216e65a03129cf6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168413456
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* cf90c3d03aee3c43b533076637a55b3fdc291769 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410)
* 318d5e5f8d22e011b21ee14e770a5a9849334b7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1640275918
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,13 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ // If old file is written with legacy mode, read with legacy mode, even if now it's non-legacy mode in conf.
+ // Since by changing the property users want to control how files are written, not how they are read.
+ // Later the value of WRITE_OLD_LIST_STRUCTURE will be picked from conf, thus either keeping old mode or writing new mode.
+ if (legacyMode) {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
Review Comment:
I think indeed this change requires thoroughful check, I believe experts in this code can find a better solution. At least the test I've written should test the potential other scenarios (for any fix being developed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2169016844
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 242cc9c34dbb23192c0c1e5d42381835c544b3e4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2173432389
@yihua @CTTY is there anything I can do more to get this reviewed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2167884837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e2086f8a2b8298e26c3068e07e6016f93f2c98a4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2176093552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
}, {
"hash" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445",
"triggerID" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"triggerType" : "PUSH"
}, {
"hash" : "c814458b48d8a33a2b5ebbb0355183e129be89f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c814458b48d8a33a2b5ebbb0355183e129be89f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 0dbffbf92f2cb18861621be3e216e65a03129cf6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445)
* c814458b48d8a33a2b5ebbb0355183e129be89f4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2176234693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
}, {
"hash" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24445",
"triggerID" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"triggerType" : "PUSH"
}, {
"hash" : "c814458b48d8a33a2b5ebbb0355183e129be89f4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24449",
"triggerID" : "c814458b48d8a33a2b5ebbb0355183e129be89f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* c814458b48d8a33a2b5ebbb0355183e129be89f4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168880209
Created https://github.com/apache/hudi/pull/11461/files
About the previous comment - I'm more interested to test 0.14.1 branch with this change - I'll try to create it if CI will run tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168862799
> > @VitoMakarevich could you also raise the same set of changes against [branch-0.x](https://github.com/apache/hudi/tree/branch-0.x), where future 0. x releases are based off? It would be good to run through the CI in `branch-0.x` too.
>
> Working on this, had to rewrite some tests with other primitives available for the previous state,and will now run tests locally and push, would it make sense to make another PR into 0.14.1 directly? I mean - we are now blocked from using 0.14.1, so this is my first priority, once it's done, I will be more calm to proceed with merging into the main.
We cut 0.x releases from `branch-0.x`, so it makes sense to raise another PR on `branch-0.x` (CI will be triggered for such a PR). Running tests in `branch-0.x` still gives you confidence since `branch-0.x` only has minor changes on top of release 0.14.1 on this code path. We freeze release branches, e.g., 0.14.1, so you may not be able to merge PR to 0.14.1 release branch; Nevertheless, you can still main your fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166919410
Also I see [PR](https://github.com/apache/hudi/pull/7512) which introduced custom class of
`hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java`.
As I understand it came after the developers' decision to not use the schema with which the file has been written in favor of the deduced writer schema. So the purpose of the previous PR was that:
If for some reason Parquet file is written in a new style(3-level nesting) - likely with some other than Spark tool or with "spark.hadoop.parquet.avro.write-old-list-structure", "false" - then if there is no overrides(safety measures first), kindly request reader to read it as a new style.
Without it - it was likely leading to the same issue we are facing, so the current change is basically the reverse case - if the file was written as 2 level, no matter what setting is in the runtime, use 2 level readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168120157
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec5dda2f96699fde05db8042ab8fc45fcbb88738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405)
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 2a19d81bf1116a341380ce7d7511fbdc1f64ba81 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166917289
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166923627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383)
* 01581df81e33432179d9dfe574f8e9ae74f18038 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1639752118
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieAvroParquetReader.java:
##########
@@ -166,7 +166,7 @@ private ClosableIterator<IndexedRecord> getIndexedRecordIteratorInternal(Schema
// NOTE: We have to set both Avro read-schema and projection schema to make
// sure that in case the file-schema is not equal to read-schema we'd still
// be able to read that file (in case projection is a proper one)
- Configuration hadoopConf = storage.getConf().unwrapAs(Configuration.class);
+ Configuration hadoopConf = storage.getConf().unwrapCopyAs(Configuration.class);
Review Comment:
I'm not sure about this line, but since we forcefully need to override properties, I'm making a copy here for the reader, so global conf will not be polluted with those overrides we add later in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1639751103
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,10 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ if (legacyMode) {
Review Comment:
Done.
Added 10 potential scenarios,
marked those 4 that failed with the previous code and added comments around:
1. 2 are leading to exceptions we receive
2. 2 are leading to silent dataloss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168103198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec5dda2f96699fde05db8042ab8fc45fcbb88738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405)
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1639853983
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -46,11 +46,7 @@ public HoodieAvroReadSupport() {
@Override
public ReadContext init(Configuration configuration, Map<String, String> keyValueMetaData, MessageType fileSchema) {
boolean legacyMode = checkLegacyMode(fileSchema.getFields());
- // support non-legacy list
- if (!legacyMode && configuration.get(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE) == null) {
Review Comment:
Essentially you can reproduce it if run with the old code - scenarios 5 and 6 in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2167021098
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 01581df81e33432179d9dfe574f8e9ae74f18038 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2170538719
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 469138465aa8110f8c1478914d09e286fdd9f2d7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417)
* a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1642969084
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -62,6 +58,23 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
return new ReadContext(requestedSchema, readContext.getReadSupportMetadata());
}
+ /**
+ * Here we want set config with which file has been written.
+ * Even though user may have overwritten {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE},
+ * it's only applicable to how to produce new files(here is a read path).
+ * Later the config value {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE} will still be used
+ * to write new file according to the user preferences.
+ **/
+ private void adjustConfToReadWithFileProduceMode(Boolean isLegacyModeWrittenFile, Configuration configuration) {
+ if (isLegacyModeWrittenFile) {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
+ "true", "support reading avro from legacy map/list in parquet file");
+ } else {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
+ "false", "support reading avro from non-legacy map/list in parquet file");
+ }
Review Comment:
I would say the most important would be to preserve compatibility with previous versions of Hudi - so it should be able to normally read both versions (2 level and 3 level)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2174607337
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421)
* 93ddf4151fedb0698e0c5c56d69b4b866626d393 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2176266302
> Also I see that this block
>
> ```
> if (!legacyMode) {
> requestedSchema = new MessageType(requestedSchema.getName(), convertLegacyMap(requestedSchema.getFields()));
> }
> ```
>
> is redundant - since in all the cases `requestedSchema` fetched after tuning AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE equals to schema coming from this block - meaning AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE does the correct conversion. So this code block should be removed to not cause confusion
Sorry -this is incorrect, it looks like Spark 3.1 fails without this block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166758114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1640398375
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,10 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ if (legacyMode) {
Review Comment:
Great. What I meant is that new tests should be added, which fail before this fix and succeed after this fix, which is what you're doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1638823816
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,10 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ if (legacyMode) {
Review Comment:
Is it possible to write a unit test so that the test fails before this fix and succeeds after the fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168244583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 2a19d81bf1116a341380ce7d7511fbdc1f64ba81 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406)
* 836c87e014c4f94657d5303d08c3e6ea5794fc72 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407)
* efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168316774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409)
* cf90c3d03aee3c43b533076637a55b3fdc291769 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168695390
If you want the very exact part where it fails - please read https://apache-hudi.slack.com/archives/C4D716NPQ/p1718331551975499?thread_ts=1718277342.110129&cid=C4D716NPQ
Probably putting a breakpoint into the code Yauhen mentioned should give you a very clear understanding of why when the structure changes, fields are read as `NULL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2169035741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 242cc9c34dbb23192c0c1e5d42381835c544b3e4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415)
* 469138465aa8110f8c1478914d09e286fdd9f2d7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1643339632
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -62,6 +58,23 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
return new ReadContext(requestedSchema, readContext.getReadSupportMetadata());
}
+ /**
+ * Here we want set config with which file has been written.
+ * Even though user may have overwritten {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE},
+ * it's only applicable to how to produce new files(here is a read path).
+ * Later the config value {@link AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE} will still be used
+ * to write new file according to the user preferences.
+ **/
+ private void adjustConfToReadWithFileProduceMode(Boolean isLegacyModeWrittenFile, Configuration configuration) {
Review Comment:
Does this look fine besides this minor change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166774450
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168840785
Pasting Yauhen's comment here so we have the context:
To elaborate a bit more on a possible silent data loss scenario we discovered, but in a general way. In parquet-hadoop `ParquetFileReader#internalReadRowGroup` there is a code
```
for (ColumnChunkMetaData mc : block.getColumns()) {
ColumnPath pathKey = mc.getPath();
ColumnDescriptor columnDescriptor = paths.get(pathKey);
if (columnDescriptor != null) {
```
Here `block` is constructed from the parquet's footer and if read from the legacy list format file will have entry similar to this
```
{IntColumnChunkMetaData@22604} "ColumnMetaData{GZIP [internal_list, array] repeated int64 array [PLAIN_DICTIONARY, RLE], 643}"
```
However, `paths` which initially are correctly built from the parquet's footer will be mutated with writer's schema in `HoodieAvroReadSupport` during merge logic in `HoodieMergeHelper`. So the map for `paths` will now contain a key/value pair like
```
{ColumnPath@24497} "[internal_list, list, element]" -> {ColumnDescriptor@24158} "[internal_list, list, element] required int64 element"
```
Therefore, `columnDescriptor = paths.get(pathKey)` will return null from the java map because pathKey [internal_list, array] could not be found. So `internalReadRowGroup` will return `rowGroup` with all values populated except `internal_list` which will be completely missed out and end up having the null value in the hoodie record which then will be written to a new parquet file (if the writer schema allows optional value)
This behaviour is perfectly acceptable for the new nullable columns but for the columns that should hold the same compatibility semantics while changing the path in the context of `ParquetFileReader` can lead to a subtle data loss issues like we see in that two/three level list case
I cannot think of the general robust way to handle it outside of parquet-hadoop code but there might be a room for an additional validation logic when `requestedSchema` is constructed in `HoodieAvroReadSupport` and `super.init` is called
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1640400365
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/HoodieAvroParquetReader.java:
##########
@@ -166,7 +166,7 @@ private ClosableIterator<IndexedRecord> getIndexedRecordIteratorInternal(Schema
// NOTE: We have to set both Avro read-schema and projection schema to make
// sure that in case the file-schema is not equal to read-schema we'd still
// be able to read that file (in case projection is a proper one)
- Configuration hadoopConf = storage.getConf().unwrapAs(Configuration.class);
+ Configuration hadoopConf = storage.getConf().unwrapCopyAs(Configuration.class);
Review Comment:
Yes, if you're changing the storage configuration for this reader use only, it's better to make a copy before any change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168837738
@jonvex could you review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2174597093
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421)
* 93ddf4151fedb0698e0c5c56d69b4b866626d393 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2174714087
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 93ddf4151fedb0698e0c5c56d69b4b866626d393 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2175709698
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
}, {
"hash" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24415",
"triggerID" : "242cc9c34dbb23192c0c1e5d42381835c544b3e4",
"triggerType" : "PUSH"
}, {
"hash" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24417",
"triggerID" : "469138465aa8110f8c1478914d09e286fdd9f2d7",
"triggerType" : "PUSH"
}, {
"hash" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24421",
"triggerID" : "a9bf8defb7dec0163a9f751ae757c5cc8ed17e2d",
"triggerType" : "PUSH"
}, {
"hash" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440",
"triggerID" : "93ddf4151fedb0698e0c5c56d69b4b866626d393",
"triggerType" : "PUSH"
}, {
"hash" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0dbffbf92f2cb18861621be3e216e65a03129cf6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 93ddf4151fedb0698e0c5c56d69b4b866626d393 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24440)
* 0dbffbf92f2cb18861621be3e216e65a03129cf6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2167988483
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e2086f8a2b8298e26c3068e07e6016f93f2c98a4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402)
* ec5dda2f96699fde05db8042ab8fc45fcbb88738 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1639851878
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -46,11 +46,7 @@ public HoodieAvroReadSupport() {
@Override
public ReadContext init(Configuration configuration, Map<String, String> keyValueMetaData, MessageType fileSchema) {
boolean legacyMode = checkLegacyMode(fileSchema.getFields());
- // support non-legacy list
- if (!legacyMode && configuration.get(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE) == null) {
Review Comment:
This line has been extracted and now I'm forcefully applying configuration even if something is set by the user, since if the user has `AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE=true`, then the case with rewriting 3 levels into 2 levels, read will return NULL, so at the end either fail/silent data loss at the writing part - these are scenarios 5 and 6 from the test suite.
This is opposite use case as we faced initially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #11450:
URL: https://github.com/apache/hudi/pull/11450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166794881
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1638937534
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,10 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ if (legacyMode) {
Review Comment:
Yes, struggling to do this with all Hudi test rails, but slowly proceeding.
I'm not sure exactly understand what you mean since if I have this code block it will anyway proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1638823816
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,10 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ if (legacyMode) {
Review Comment:
Is it possible to write a simple test so that the test fails before this fix and succeeds after the fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2166852778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a13f012152b0cd41feccfd589ea438f3e1697607 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2167870875
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 01581df81e33432179d9dfe574f8e9ae74f18038 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391)
* e2086f8a2b8298e26c3068e07e6016f93f2c98a4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168507387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 318d5e5f8d22e011b21ee14e770a5a9849334b7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412)
* dbff5f176b79f5848598a3041540b2b965aafa2c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "VitoMakarevich (via GitHub)" <gi...@apache.org>.
VitoMakarevich commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1640356236
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,10 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ if (legacyMode) {
Review Comment:
Added generation of those scenarios possible in theory(omitted some that require a lot of bootstrapping).
Also left separately 4 aforementioned scenarios, so the developer will be able to run them separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on code in PR #11450:
URL: https://github.com/apache/hudi/pull/11450#discussion_r1639160519
##########
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java:
##########
@@ -51,6 +51,13 @@ public ReadContext init(Configuration configuration, Map<String, String> keyValu
configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
"false", "support reading avro from non-legacy map/list in parquet file");
}
+ // If old file is written with legacy mode, read with legacy mode, even if now it's non-legacy mode in conf.
+ // Since by changing the property users want to control how files are written, not how they are read.
+ // Later the value of WRITE_OLD_LIST_STRUCTURE will be picked from conf, thus either keeping old mode or writing new mode.
+ if (legacyMode) {
+ configuration.set(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
Review Comment:
look like user has set this conf `false` to controller the action, we'd better not change user behavior. just like`datetimeRebaseModeInWrite`, I don't want compatible history when I set `EXCEPTION`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168135700
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec5dda2f96699fde05db8042ab8fc45fcbb88738 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405)
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 2a19d81bf1116a341380ce7d7511fbdc1f64ba81 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406)
* 836c87e014c4f94657d5303d08c3e6ea5794fc72 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168330311
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409)
* cf90c3d03aee3c43b533076637a55b3fdc291769 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7874] Fix Hudi being able to read 2-level structure [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11450:
URL: https://github.com/apache/hudi/pull/11450#issuecomment-2168519223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"triggerType" : "PUSH"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166774450",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166784596",
"triggerType" : "MANUAL"
}, {
"hash" : "a13f012152b0cd41feccfd589ea438f3e1697607",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24383",
"triggerID" : "2166879255",
"triggerType" : "MANUAL"
}, {
"hash" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24391",
"triggerID" : "01581df81e33432179d9dfe574f8e9ae74f18038",
"triggerType" : "PUSH"
}, {
"hash" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24402",
"triggerID" : "e2086f8a2b8298e26c3068e07e6016f93f2c98a4",
"triggerType" : "PUSH"
}, {
"hash" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24405",
"triggerID" : "ec5dda2f96699fde05db8042ab8fc45fcbb88738",
"triggerType" : "PUSH"
}, {
"hash" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f876303cc2ed8f203f8db9f3dea972e3a28f0b7",
"triggerType" : "PUSH"
}, {
"hash" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24406",
"triggerID" : "2a19d81bf1116a341380ce7d7511fbdc1f64ba81",
"triggerType" : "PUSH"
}, {
"hash" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24407",
"triggerID" : "836c87e014c4f94657d5303d08c3e6ea5794fc72",
"triggerType" : "PUSH"
}, {
"hash" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24408",
"triggerID" : "91620c7b073209cc0cd7c70165fa2f4231112e44",
"triggerType" : "PUSH"
}, {
"hash" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24409",
"triggerID" : "efdc2ebd02b95e39eacfcae0e7e68fd0e16a7495",
"triggerType" : "PUSH"
}, {
"hash" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24410",
"triggerID" : "cf90c3d03aee3c43b533076637a55b3fdc291769",
"triggerType" : "PUSH"
}, {
"hash" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412",
"triggerID" : "318d5e5f8d22e011b21ee14e770a5a9849334b7e",
"triggerType" : "PUSH"
}, {
"hash" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414",
"triggerID" : "dbff5f176b79f5848598a3041540b2b965aafa2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f876303cc2ed8f203f8db9f3dea972e3a28f0b7 UNKNOWN
* 318d5e5f8d22e011b21ee14e770a5a9849334b7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24412)
* dbff5f176b79f5848598a3041540b2b965aafa2c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org