You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2024/05/06 18:03:57 UTC
[PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
codope opened a new pull request, #11162:
URL: https://github.com/apache/hudi/pull/11162
### Change Logs
This is stacked on top of #11077 and #11146. This PR integrates the secondary index on the reader path as follows:
- Add `HoodieTableMetadata#readSecondaryIndex` API
- Add `SecondaryIndexSupport`, a util class to get candidate files based on query filter and secondary index
- Changes in `HoodieFileIndex` to use `SecondaryIndexSupport`
### Impact
Can use secondary index in queries.
### Risk level (write none, low medium or high below)
medium
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2096659948
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7952ad87456d56a737eb43813807d64551ddc02b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2148132863
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* ab2875d506fbb642636ca10d044fa9b9e5c951ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203)
* c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2154190933
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
}, {
"hash" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249",
"triggerID" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"triggerType" : "PUSH"
}, {
"hash" : "db219f61a2efd1b601ba1ec14f7e2871df5d615d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24265",
"triggerID" : "db219f61a2efd1b601ba1ec14f7e2871df5d615d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* db219f61a2efd1b601ba1ec14f7e2871df5d615d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24265)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2145748432
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* b00831e0a0506714d27bc2a64e58084b357a83cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196)
* ab2875d506fbb642636ca10d044fa9b9e5c951ae Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2154064293
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
}, {
"hash" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249",
"triggerID" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"triggerType" : "PUSH"
}, {
"hash" : "db219f61a2efd1b601ba1ec14f7e2871df5d615d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db219f61a2efd1b601ba1ec14f7e2871df5d615d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249)
* db219f61a2efd1b601ba1ec14f7e2871df5d615d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2150795998
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* 8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2120260409
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c2c253f3316324240407570d8f1e063773bddd7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003)
* 96b10a6c9a2457512cbe8bb33bb974496aa515c0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2140488986
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a602c9c4234062e66877fc4bf2c50f94f43767bc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142)
* 3c52961bdbcb210e4c7140f5939143cfda7adb50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2150782235
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* 7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232)
* 8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2151705351
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* 8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233)
* 19b6362eb1d0e7c2ea15465964bef441b441106f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2151720640
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* 8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233)
* 19b6362eb1d0e7c2ea15465964bef441b441106f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2152811360
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* bb3bff3f12c7472714f86aad930737238c1a310d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2097544471
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7952ad87456d56a737eb43813807d64551ddc02b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712)
* 8e50a635f9dad16057b0c35121716712856a8511 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2099054722
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e50a635f9dad16057b0c35121716712856a8511 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730)
* 49f3c3fb9f2d802b49b1553db7285e4310faefe4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137483472
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d5e22743cf22eba8476470a07876561577b0d8f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119)
* 1cdba2db5a4a61458263e31b7306c91ba7c79196 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1624004588
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SecondaryIndexSupport.scala:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.RecordLevelIndexSupport.filterQueryWithRecordKey
+import org.apache.hudi.SecondaryIndexSupport.filterQueriesWithSecondaryKey
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.FileSlice
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.metadata.HoodieTableMetadataUtil.PARTITION_NAME_SECONDARY_INDEX
+import org.apache.hudi.storage.StoragePath
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+import scala.collection.JavaConverters._
+import scala.collection.{JavaConverters, mutable}
+
+class SecondaryIndexSupport(spark: SparkSession,
+ metadataConfig: HoodieMetadataConfig,
+ metaClient: HoodieTableMetaClient) extends RecordLevelIndexSupport(spark, metadataConfig, metaClient) {
+ override def getIndexName: String = SecondaryIndexSupport.INDEX_NAME
Review Comment:
This is also a code reuse from RLI right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1624769566
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -851,6 +852,158 @@ private Map<String, String> reverseLookupSecondaryKeys(String partitionName, Lis
return recordKeyMap;
}
+ @Override
+ protected Map<String, List<HoodieRecord<HoodieMetadataPayload>>> getSecondaryIndexRecords(List<String> keys, String partitionName) {
+ if (keys.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> result = new HashMap<>();
+
+ // Load the file slices for the partition. Each file slice is a shard which saves a portion of the keys.
+ List<FileSlice> partitionFileSlices = partitionFileSliceMap.computeIfAbsent(partitionName,
+ k -> HoodieTableMetadataUtil.getPartitionLatestMergedFileSlices(metadataMetaClient, metadataFileSystemView, partitionName));
+ final int numFileSlices = partitionFileSlices.size();
+ ValidationUtils.checkState(numFileSlices > 0, "Number of file slices for partition " + partitionName + " should be > 0");
+
+ // Lookup keys from each file slice
+ // TODO: parallelize this loop
+ for (FileSlice partition : partitionFileSlices) {
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> currentFileSliceResult = lookupSecondaryKeysFromFileSlice(partitionName, keys, partition);
+
+ currentFileSliceResult.forEach((secondaryKey, secondaryRecords) -> {
+ result.merge(secondaryKey, secondaryRecords, (oldRecords, newRecords) -> {
+ newRecords.addAll(oldRecords);
+ return newRecords;
+ });
+ });
+ }
+
+ return result;
+ }
+
+ /**
+ * Lookup list of keys from a single file slice.
+ *
+ * @param partitionName Name of the partition
+ * @param secondaryKeys The list of secondary keys to lookup
+ * @param fileSlice The file slice to read
+ * @return A {@code Map} of secondary-key to list of {@code HoodieRecord} for the secondary-keys which were found in the file slice
+ */
+ private Map<String, List<HoodieRecord<HoodieMetadataPayload>>> lookupSecondaryKeysFromFileSlice(String partitionName, List<String> secondaryKeys, FileSlice fileSlice) {
+ Map<String, HashMap<String, HoodieRecord>> logRecordsMap = new HashMap<>();
+
+ Pair<HoodieSeekingFileReader<?>, HoodieMetadataLogRecordReader> readers = getOrCreateReaders(partitionName, fileSlice);
+ try {
+ List<Long> timings = new ArrayList<>(1);
+ HoodieSeekingFileReader<?> baseFileReader = readers.getKey();
+ HoodieMetadataLogRecordReader logRecordScanner = readers.getRight();
+ if (baseFileReader == null && logRecordScanner == null) {
+ return Collections.emptyMap();
+ }
+
+ // Sort it here once so that we don't need to sort individually for base file and for each individual log files.
+ Set<String> secondaryKeySet = new HashSet<>(secondaryKeys.size());
+ List<String> sortedSecondaryKeys = new ArrayList<>(secondaryKeys);
+ Collections.sort(sortedSecondaryKeys);
Review Comment:
Good point! I have now parallelized the lookup through engineContext. So, this sorting would only be limited to single partition of data which should not spill to disk. But, even if it does, spark will handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2150688074
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217)
* 7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137985942
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2150702446
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* 7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232)
* 8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1630650191
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SecondaryIndexSupport.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.RecordLevelIndexSupport.filterQueryWithRecordKey
+import org.apache.hudi.SecondaryIndexSupport.filterQueriesWithSecondaryKey
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.FileSlice
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.metadata.HoodieTableMetadataUtil.PARTITION_NAME_SECONDARY_INDEX
+import org.apache.hudi.storage.StoragePath
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+import scala.collection.JavaConverters._
+import scala.collection.{JavaConverters, mutable}
+
+class SecondaryIndexSupport(spark: SparkSession,
+ metadataConfig: HoodieMetadataConfig,
+ metaClient: HoodieTableMetaClient) extends RecordLevelIndexSupport(spark, metadataConfig, metaClient) {
+ override def getIndexName: String = SecondaryIndexSupport.INDEX_NAME
+
+ override def computeCandidateFileNames(fileIndex: HoodieFileIndex,
+ queryFilters: Seq[Expression],
+ queryReferencedColumns: Seq[String],
+ prunedPartitionsAndFileSlices: Seq[(Option[BaseHoodieTableFileIndex.PartitionPath], Seq[FileSlice])],
+ shouldPushDownFilesFilter: Boolean
+ ): Option[Set[String]] = {
+ val secondaryKeyConfigOpt = getSecondaryKeyConfig(queryReferencedColumns, metaClient)
+ if (secondaryKeyConfigOpt.isEmpty) {
+ Option.empty
Review Comment:
return Option.empty
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SecondaryIndexSupport.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.RecordLevelIndexSupport.filterQueryWithRecordKey
+import org.apache.hudi.SecondaryIndexSupport.filterQueriesWithSecondaryKey
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.FileSlice
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.metadata.HoodieTableMetadataUtil.PARTITION_NAME_SECONDARY_INDEX
+import org.apache.hudi.storage.StoragePath
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+import scala.collection.JavaConverters._
+import scala.collection.{JavaConverters, mutable}
+
+class SecondaryIndexSupport(spark: SparkSession,
+ metadataConfig: HoodieMetadataConfig,
+ metaClient: HoodieTableMetaClient) extends RecordLevelIndexSupport(spark, metadataConfig, metaClient) {
+ override def getIndexName: String = SecondaryIndexSupport.INDEX_NAME
+
+ override def computeCandidateFileNames(fileIndex: HoodieFileIndex,
+ queryFilters: Seq[Expression],
+ queryReferencedColumns: Seq[String],
+ prunedPartitionsAndFileSlices: Seq[(Option[BaseHoodieTableFileIndex.PartitionPath], Seq[FileSlice])],
+ shouldPushDownFilesFilter: Boolean
+ ): Option[Set[String]] = {
+ val secondaryKeyConfigOpt = getSecondaryKeyConfig(queryReferencedColumns, metaClient)
+ if (secondaryKeyConfigOpt.isEmpty) {
+ Option.empty
+ }
+ lazy val (_, secondaryKeys) = if (isIndexAvailable) filterQueriesWithSecondaryKey(queryFilters, secondaryKeyConfigOpt.map(_._2)) else (List.empty, List.empty)
+ if (isIndexAvailable && queryFilters.nonEmpty && secondaryKeys.nonEmpty) {
+ val allFiles = fileIndex.inputFiles.map(strPath => new StoragePath(strPath)).toSeq
Review Comment:
can we use `prunedPartitionsAndFileSlices` replace allFiles?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/SecondaryIndexTestBase.scala:
##########
@@ -62,4 +69,54 @@ class SecondaryIndexTestBase extends HoodieSparkClientTestBase {
cleanupResources()
}
+ def verifyQueryPredicate(hudiOpts: Map[String, String], columnName: String): Unit = {
+ mergedDfList = spark.read.format("hudi").options(hudiOpts).load(basePath).repartition(1).cache() :: mergedDfList
+ val secondaryKey = mergedDfList.last.limit(1).collect().map(row => row.getAs(columnName).toString)
+ val dataFilter = EqualTo(attribute(columnName), Literal(secondaryKey(0)))
+ verifyFilePruning(hudiOpts, dataFilter)
+ }
+
+ private def attribute(partition: String): AttributeReference = {
+ AttributeReference(partition, StringType, nullable = true)()
+ }
+
+
+ private def verifyFilePruning(opts: Map[String, String], dataFilter: Expression): Unit = {
+ // with data skipping
+ val commonOpts = opts + ("path" -> basePath)
+ metaClient = HoodieTableMetaClient.reload(metaClient)
+ var fileIndex = HoodieFileIndex(spark, metaClient, None, commonOpts, includeLogFiles = true)
+ val filteredPartitionDirectories = fileIndex.listFiles(Seq(), Seq(dataFilter))
+ val filteredFilesCount = filteredPartitionDirectories.flatMap(s => s.files).size
+ assertTrue(filteredFilesCount < getLatestDataFilesCount(opts))
Review Comment:
need a new check `assertTrue(filteredFilesCount > 0)`, in my test, it will be wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1629072945
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -229,6 +231,13 @@ public void cancelAllJobs() {
javaSparkContext.cancelAllJobs();
}
+ @Override
+ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O, I, O> seqOp, Functions.Function2<O, O, O> combOp) {
+ Function2<O, I, O> seqOpFunc = seqOp::apply;
+ Function2<O, O, O> combOpFunc = combOp::apply;
+ return HoodieJavaRDD.getJavaRDD(data).aggregate(zeroValue, seqOpFunc, combOpFunc);
Review Comment:
So you are saying the cache key could be string anyway, it's the merger that plays the role for the secondary index?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2153158313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
}, {
"hash" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249",
"triggerID" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1624001312
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -851,6 +852,158 @@ private Map<String, String> reverseLookupSecondaryKeys(String partitionName, Lis
return recordKeyMap;
}
+ @Override
+ protected Map<String, List<HoodieRecord<HoodieMetadataPayload>>> getSecondaryIndexRecords(List<String> keys, String partitionName) {
+ if (keys.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> result = new HashMap<>();
+
+ // Load the file slices for the partition. Each file slice is a shard which saves a portion of the keys.
+ List<FileSlice> partitionFileSlices = partitionFileSliceMap.computeIfAbsent(partitionName,
+ k -> HoodieTableMetadataUtil.getPartitionLatestMergedFileSlices(metadataMetaClient, metadataFileSystemView, partitionName));
+ final int numFileSlices = partitionFileSlices.size();
+ ValidationUtils.checkState(numFileSlices > 0, "Number of file slices for partition " + partitionName + " should be > 0");
+
+ // Lookup keys from each file slice
+ // TODO: parallelize this loop
+ for (FileSlice partition : partitionFileSlices) {
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> currentFileSliceResult = lookupSecondaryKeysFromFileSlice(partitionName, keys, partition);
+
+ currentFileSliceResult.forEach((secondaryKey, secondaryRecords) -> {
+ result.merge(secondaryKey, secondaryRecords, (oldRecords, newRecords) -> {
+ newRecords.addAll(oldRecords);
+ return newRecords;
+ });
+ });
+ }
+
+ return result;
+ }
+
+ /**
+ * Lookup list of keys from a single file slice.
+ *
+ * @param partitionName Name of the partition
+ * @param secondaryKeys The list of secondary keys to lookup
+ * @param fileSlice The file slice to read
+ * @return A {@code Map} of secondary-key to list of {@code HoodieRecord} for the secondary-keys which were found in the file slice
+ */
+ private Map<String, List<HoodieRecord<HoodieMetadataPayload>>> lookupSecondaryKeysFromFileSlice(String partitionName, List<String> secondaryKeys, FileSlice fileSlice) {
+ Map<String, HashMap<String, HoodieRecord>> logRecordsMap = new HashMap<>();
+
+ Pair<HoodieSeekingFileReader<?>, HoodieMetadataLogRecordReader> readers = getOrCreateReaders(partitionName, fileSlice);
+ try {
+ List<Long> timings = new ArrayList<>(1);
+ HoodieSeekingFileReader<?> baseFileReader = readers.getKey();
+ HoodieMetadataLogRecordReader logRecordScanner = readers.getRight();
+ if (baseFileReader == null && logRecordScanner == null) {
+ return Collections.emptyMap();
+ }
+
+ // Sort it here once so that we don't need to sort individually for base file and for each individual log files.
+ Set<String> secondaryKeySet = new HashSet<>(secondaryKeys.size());
+ List<String> sortedSecondaryKeys = new ArrayList<>(secondaryKeys);
+ Collections.sort(sortedSecondaryKeys);
Review Comment:
Wondering if we have general sort solution that supports spilling to disk, the in-memory sort is slow and also has risk of OOM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2144669166
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df12fa59cbba5b14bb98d66dffb510f5b1659177 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195)
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* b00831e0a0506714d27bc2a64e58084b357a83cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2120242139
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c2c253f3316324240407570d8f1e063773bddd7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003)
* 96b10a6c9a2457512cbe8bb33bb974496aa515c0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2119821666
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afa509c76da8cd4215d962749620fb9896f564ea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002)
* 1c2c253f3316324240407570d8f1e063773bddd7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1630522596
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -851,6 +862,167 @@ private Map<String, String> reverseLookupSecondaryKeys(String partitionName, Lis
return recordKeyMap;
}
+ @Override
+ protected Map<String, List<HoodieRecord<HoodieMetadataPayload>>> getSecondaryIndexRecords(List<String> keys, String partitionName) {
+ if (keys.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Load the file slices for the partition. Each file slice is a shard which saves a portion of the keys.
+ List<FileSlice> partitionFileSlices = partitionFileSliceMap.computeIfAbsent(partitionName,
+ k -> HoodieTableMetadataUtil.getPartitionLatestMergedFileSlices(metadataMetaClient, metadataFileSystemView, partitionName));
+ final int numFileSlices = partitionFileSlices.size();
+ ValidationUtils.checkState(numFileSlices > 0, "Number of file slices for partition " + partitionName + " should be > 0");
+
+ engineContext.setJobStatus(this.getClass().getSimpleName(), "Lookup keys from each file slice");
+ HoodieData<FileSlice> partitionRDD = engineContext.parallelize(partitionFileSlices);
+ // Define the seqOp function (merges elements within a partition)
+ Functions.Function2<Map<String, List<HoodieRecord<HoodieMetadataPayload>>>, FileSlice, Map<String, List<HoodieRecord<HoodieMetadataPayload>>>> seqOp =
+ (accumulator, partition) -> {
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> currentFileSliceResult = lookupSecondaryKeysFromFileSlice(partitionName, keys, partition);
+ currentFileSliceResult.forEach((secondaryKey, secondaryRecords) -> accumulator.merge(secondaryKey, secondaryRecords, (oldRecords, newRecords) -> {
+ newRecords.addAll(oldRecords);
+ return newRecords;
+ }));
+ return accumulator;
+ };
+ // Define the combOp function (merges elements across partitions)
+ Functions.Function2<Map<String, List<HoodieRecord<HoodieMetadataPayload>>>, Map<String, List<HoodieRecord<HoodieMetadataPayload>>>, Map<String, List<HoodieRecord<HoodieMetadataPayload>>>> combOp =
+ (map1, map2) -> {
+ map2.forEach((secondaryKey, secondaryRecords) -> map1.merge(secondaryKey, secondaryRecords, (oldRecords, newRecords) -> {
+ newRecords.addAll(oldRecords);
+ return newRecords;
+ }));
+ return map1;
+ };
+ // Use aggregate to merge results within and across partitions
+ // Define the zero value (initial value)
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> zeroValue = new HashMap<>();
+ return engineContext.aggregate(partitionRDD, zeroValue, seqOp, combOp);
+ }
+
+ /**
+ * Lookup list of keys from a single file slice.
+ *
+ * @param partitionName Name of the partition
+ * @param secondaryKeys The list of secondary keys to lookup
+ * @param fileSlice The file slice to read
+ * @return A {@code Map} of secondary-key to list of {@code HoodieRecord} for the secondary-keys which were found in the file slice
+ */
+ private Map<String, List<HoodieRecord<HoodieMetadataPayload>>> lookupSecondaryKeysFromFileSlice(String partitionName, List<String> secondaryKeys, FileSlice fileSlice) {
+ Map<String, HashMap<String, HoodieRecord>> logRecordsMap = new HashMap<>();
+
+ Pair<HoodieSeekingFileReader<?>, HoodieMetadataLogRecordReader> readers = getOrCreateReaders(partitionName, fileSlice);
+ try {
+ List<Long> timings = new ArrayList<>(1);
+ HoodieSeekingFileReader<?> baseFileReader = readers.getKey();
+ HoodieMetadataLogRecordReader logRecordScanner = readers.getRight();
+ if (baseFileReader == null && logRecordScanner == null) {
+ return Collections.emptyMap();
+ }
+
+ // Sort it here once so that we don't need to sort individually for base file and for each individual log files.
+ Set<String> secondaryKeySet = new HashSet<>(secondaryKeys.size());
+ List<String> sortedSecondaryKeys = new ArrayList<>(secondaryKeys);
+ Collections.sort(sortedSecondaryKeys);
+ secondaryKeySet.addAll(sortedSecondaryKeys);
+
+ logRecordScanner.getRecords().forEach(record -> {
+ HoodieMetadataPayload payload = record.getData();
+ String recordKey = payload.getRecordKeyFromSecondaryIndex();
+ if (secondaryKeySet.contains(recordKey)) {
+ String secondaryKey = payload.getRecordKeyFromSecondaryIndex();
+ logRecordsMap.computeIfAbsent(secondaryKey, k -> new HashMap<>()).put(recordKey, record);
+ }
+ });
+
+ return readNonUniqueRecordsAndMergeWithLogRecords(baseFileReader, sortedSecondaryKeys, logRecordsMap, timings, partitionName);
+ } catch (IOException ioe) {
+ throw new HoodieIOException("Error merging records from metadata table for " + secondaryKeys.size() + " key : ", ioe);
+ } finally {
+ if (!reuse) {
+ closeReader(readers);
+ }
+ }
+ }
+
+ private Map<String, List<HoodieRecord<HoodieMetadataPayload>>> readNonUniqueRecordsAndMergeWithLogRecords(HoodieSeekingFileReader<?> reader,
+ List<String> sortedKeys,
+ Map<String, HashMap<String, HoodieRecord>> logRecordsMap,
+ List<Long> timings,
+ String partitionName) throws IOException {
+ HoodieTimer timer = HoodieTimer.start();
+
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> resultMap = new HashMap<>();
+ if (reader == null) {
+ // No base file at all
+ timings.add(timer.endTimer());
+ logRecordsMap.forEach((secondaryKey, logRecords) -> {
+ List<HoodieRecord<HoodieMetadataPayload>> recordList = new ArrayList<>();
+ logRecords.values().forEach(record -> {
+ recordList.add((HoodieRecord<HoodieMetadataPayload>)record);
+ });
+ resultMap.put(secondaryKey, recordList);
+ });
+ return resultMap;
+ }
+
+ HoodieTimer readTimer = HoodieTimer.start();
+
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> baseFileRecordsMap =
+ fetchBaseFileAllRecordsByKeys(reader, sortedKeys, true, partitionName);
+ logRecordsMap.forEach((secondaryKey, logRecords) -> {
+ if (!baseFileRecordsMap.containsKey(secondaryKey)) {
+ List<HoodieRecord<HoodieMetadataPayload>> recordList = logRecords
+ .values()
+ .stream()
+ .map(record -> (HoodieRecord<HoodieMetadataPayload>) record)
+ .collect(Collectors.toList());
+
+ resultMap.put(secondaryKey, recordList);
+ } else {
+ List<HoodieRecord<HoodieMetadataPayload>> baseFileRecords = baseFileRecordsMap.get(secondaryKey);
+ List<HoodieRecord<HoodieMetadataPayload>> resultRecords = new ArrayList<>();
+
+ baseFileRecords.forEach(prevRecord -> {
+ HoodieMetadataPayload prevPayload = prevRecord.getData();
+ String recordKey = prevPayload.getRecordKeyFromSecondaryIndex();
+
+ if (!logRecords.containsKey(recordKey)) {
+ resultRecords.add(prevRecord);
+ } else {
+ // Merge the records
+ HoodieRecord<HoodieMetadataPayload> newRecord = logRecords.get(recordKey);
+ HoodieMetadataPayload newPayload = newRecord.getData();
+
+ assert recordKey.equals(newPayload.getRecordKeyFromSecondaryIndex());
Review Comment:
We should never have assertion in production code, let's use `ValidationUtils` and add comprehensive prompt for the check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1624781058
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -229,6 +231,13 @@ public void cancelAllJobs() {
javaSparkContext.cancelAllJobs();
}
+ @Override
+ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O, I, O> seqOp, Functions.Function2<O, O, O> combOp) {
+ Function2<O, I, O> seqOpFunc = seqOp::apply;
+ Function2<O, O, O> combOpFunc = combOp::apply;
+ return HoodieJavaRDD.getJavaRDD(data).aggregate(zeroValue, seqOpFunc, combOpFunc);
Review Comment:
This is based on https://github.com/apache/spark/blob/7e8b60b5ae7d6453bc1ce51b5112c975f9aa8757/core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala#L426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2144440444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c52961bdbcb210e4c7140f5939143cfda7adb50 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151)
* df12fa59cbba5b14bb98d66dffb510f5b1659177 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1628207181
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataLogRecordReader.java:
##########
@@ -253,7 +253,7 @@ public HoodieMetadataLogRecordReader build() {
}
private boolean shouldUseMetadataMergedLogRecordScanner() {
- return PARTITION_NAME_SECONDARY_INDEX.equals(partitionName);
+ return partitionName.startsWith(PARTITION_NAME_SECONDARY_INDEX_PREFIX);
Review Comment:
note to reviewer: this is amin fix in the latest commit. The issue was not caught in the testing previously because I was only asserting count of records. I have improved the test. Now, we assert the secondary index records as well as check that file pruning happens. Please see `TestSecondaryIndexPruning`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/SecondaryIndexTestBase.scala:
##########
@@ -62,4 +69,54 @@ class SecondaryIndexTestBase extends HoodieSparkClientTestBase {
cleanupResources()
}
+ def verifyQueryPredicate(hudiOpts: Map[String, String], columnName: String): Unit = {
+ mergedDfList = spark.read.format("hudi").options(hudiOpts).load(basePath).repartition(1).cache() :: mergedDfList
+ val secondaryKey = mergedDfList.last.limit(1).collect().map(row => row.getAs(columnName).toString)
+ val dataFilter = EqualTo(attribute(columnName), Literal(secondaryKey(0)))
+ verifyFilePruning(hudiOpts, dataFilter)
+ }
+
+ private def attribute(partition: String): AttributeReference = {
+ AttributeReference(partition, StringType, nullable = true)()
+ }
+
+
+ private def verifyFilePruning(opts: Map[String, String], dataFilter: Expression): Unit = {
Review Comment:
note to reviewer: the method here and below just help with verifying file pruning happens correctly i.e. with data skipping enabled, filtered file count < data files in latest snapshot, and with data skipping disabled they should be equal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2148146035
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* ab2875d506fbb642636ca10d044fa9b9e5c951ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203)
* c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2097535475
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7952ad87456d56a737eb43813807d64551ddc02b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712)
* 8e50a635f9dad16057b0c35121716712856a8511 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1623995661
##########
hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java:
##########
@@ -312,6 +312,33 @@ public Map<String, List<HoodieRecordGlobalLocation>> readRecordIndex(List<String
return recordKeyToLocation;
}
+ /**
+ * Get record-location using secondary-index and record-index
+ * <p>
+ * If the Metadata Table is not enabled, an exception is thrown to distinguish this from the absence of the key.
+ *
+ * @param secondaryKeys The list of secondary keys to read
+ */
+ @Override
+ public Map<String, List<HoodieRecordGlobalLocation>> readSecondaryIndex(List<String> secondaryKeys) {
+ ValidationUtils.checkState(dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.RECORD_INDEX),
+ "Record index is not initialized in MDT");
+ ValidationUtils.checkState(
+ dataMetaClient.getTableConfig().getMetadataPartitions().stream().anyMatch(partitionName -> partitionName.startsWith(MetadataPartitionType.SECONDARY_INDEX.getPartitionPath())),
+ "Secondary index is not initialized in MDT");
+ // Fetch secondary-index records
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> secondaryKeyRecords = getSecondaryIndexRecords(secondaryKeys, MetadataPartitionType.SECONDARY_INDEX.getPartitionPath());
+ // Now collect the record-keys and fetch the RLI records
Review Comment:
do you mean secondary index records?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1625209429
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -229,6 +231,13 @@ public void cancelAllJobs() {
javaSparkContext.cancelAllJobs();
}
+ @Override
+ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O, I, O> seqOp, Functions.Function2<O, O, O> combOp) {
+ Function2<O, I, O> seqOpFunc = seqOp::apply;
+ Function2<O, O, O> combOpFunc = combOp::apply;
+ return HoodieJavaRDD.getJavaRDD(data).aggregate(zeroValue, seqOpFunc, combOpFunc);
Review Comment:
I didn't see changes for `HoodieMetadataMergedLogRecordScanner` that switches from string cache key to serializable, then how we support the non-string secondary index fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137739521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cdba2db5a4a61458263e31b7306c91ba7c79196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2148333915
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1630642711
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:
##########
@@ -851,6 +862,167 @@ private Map<String, String> reverseLookupSecondaryKeys(String partitionName, Lis
return recordKeyMap;
}
+ @Override
+ protected Map<String, List<HoodieRecord<HoodieMetadataPayload>>> getSecondaryIndexRecords(List<String> keys, String partitionName) {
+ if (keys.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Load the file slices for the partition. Each file slice is a shard which saves a portion of the keys.
+ List<FileSlice> partitionFileSlices = partitionFileSliceMap.computeIfAbsent(partitionName,
+ k -> HoodieTableMetadataUtil.getPartitionLatestMergedFileSlices(metadataMetaClient, metadataFileSystemView, partitionName));
+ final int numFileSlices = partitionFileSlices.size();
+ ValidationUtils.checkState(numFileSlices > 0, "Number of file slices for partition " + partitionName + " should be > 0");
+
+ engineContext.setJobStatus(this.getClass().getSimpleName(), "Lookup keys from each file slice");
+ HoodieData<FileSlice> partitionRDD = engineContext.parallelize(partitionFileSlices);
+ // Define the seqOp function (merges elements within a partition)
+ Functions.Function2<Map<String, List<HoodieRecord<HoodieMetadataPayload>>>, FileSlice, Map<String, List<HoodieRecord<HoodieMetadataPayload>>>> seqOp =
+ (accumulator, partition) -> {
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> currentFileSliceResult = lookupSecondaryKeysFromFileSlice(partitionName, keys, partition);
+ currentFileSliceResult.forEach((secondaryKey, secondaryRecords) -> accumulator.merge(secondaryKey, secondaryRecords, (oldRecords, newRecords) -> {
+ newRecords.addAll(oldRecords);
+ return newRecords;
+ }));
+ return accumulator;
+ };
+ // Define the combOp function (merges elements across partitions)
+ Functions.Function2<Map<String, List<HoodieRecord<HoodieMetadataPayload>>>, Map<String, List<HoodieRecord<HoodieMetadataPayload>>>, Map<String, List<HoodieRecord<HoodieMetadataPayload>>>> combOp =
+ (map1, map2) -> {
+ map2.forEach((secondaryKey, secondaryRecords) -> map1.merge(secondaryKey, secondaryRecords, (oldRecords, newRecords) -> {
+ newRecords.addAll(oldRecords);
+ return newRecords;
+ }));
+ return map1;
+ };
+ // Use aggregate to merge results within and across partitions
+ // Define the zero value (initial value)
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> zeroValue = new HashMap<>();
+ return engineContext.aggregate(partitionRDD, zeroValue, seqOp, combOp);
+ }
+
+ /**
+ * Lookup list of keys from a single file slice.
+ *
+ * @param partitionName Name of the partition
+ * @param secondaryKeys The list of secondary keys to lookup
+ * @param fileSlice The file slice to read
+ * @return A {@code Map} of secondary-key to list of {@code HoodieRecord} for the secondary-keys which were found in the file slice
+ */
+ private Map<String, List<HoodieRecord<HoodieMetadataPayload>>> lookupSecondaryKeysFromFileSlice(String partitionName, List<String> secondaryKeys, FileSlice fileSlice) {
+ Map<String, HashMap<String, HoodieRecord>> logRecordsMap = new HashMap<>();
+
+ Pair<HoodieSeekingFileReader<?>, HoodieMetadataLogRecordReader> readers = getOrCreateReaders(partitionName, fileSlice);
+ try {
+ List<Long> timings = new ArrayList<>(1);
+ HoodieSeekingFileReader<?> baseFileReader = readers.getKey();
+ HoodieMetadataLogRecordReader logRecordScanner = readers.getRight();
+ if (baseFileReader == null && logRecordScanner == null) {
+ return Collections.emptyMap();
+ }
+
+ // Sort it here once so that we don't need to sort individually for base file and for each individual log files.
+ Set<String> secondaryKeySet = new HashSet<>(secondaryKeys.size());
+ List<String> sortedSecondaryKeys = new ArrayList<>(secondaryKeys);
+ Collections.sort(sortedSecondaryKeys);
+ secondaryKeySet.addAll(sortedSecondaryKeys);
+
+ logRecordScanner.getRecords().forEach(record -> {
+ HoodieMetadataPayload payload = record.getData();
+ String recordKey = payload.getRecordKeyFromSecondaryIndex();
+ if (secondaryKeySet.contains(recordKey)) {
+ String secondaryKey = payload.getRecordKeyFromSecondaryIndex();
+ logRecordsMap.computeIfAbsent(secondaryKey, k -> new HashMap<>()).put(recordKey, record);
+ }
+ });
+
+ return readNonUniqueRecordsAndMergeWithLogRecords(baseFileReader, sortedSecondaryKeys, logRecordsMap, timings, partitionName);
+ } catch (IOException ioe) {
+ throw new HoodieIOException("Error merging records from metadata table for " + secondaryKeys.size() + " key : ", ioe);
+ } finally {
+ if (!reuse) {
+ closeReader(readers);
+ }
+ }
+ }
+
+ private Map<String, List<HoodieRecord<HoodieMetadataPayload>>> readNonUniqueRecordsAndMergeWithLogRecords(HoodieSeekingFileReader<?> reader,
+ List<String> sortedKeys,
+ Map<String, HashMap<String, HoodieRecord>> logRecordsMap,
+ List<Long> timings,
+ String partitionName) throws IOException {
+ HoodieTimer timer = HoodieTimer.start();
+
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> resultMap = new HashMap<>();
+ if (reader == null) {
+ // No base file at all
+ timings.add(timer.endTimer());
+ logRecordsMap.forEach((secondaryKey, logRecords) -> {
+ List<HoodieRecord<HoodieMetadataPayload>> recordList = new ArrayList<>();
+ logRecords.values().forEach(record -> {
+ recordList.add((HoodieRecord<HoodieMetadataPayload>)record);
+ });
+ resultMap.put(secondaryKey, recordList);
+ });
+ return resultMap;
+ }
+
+ HoodieTimer readTimer = HoodieTimer.start();
+
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> baseFileRecordsMap =
+ fetchBaseFileAllRecordsByKeys(reader, sortedKeys, true, partitionName);
+ logRecordsMap.forEach((secondaryKey, logRecords) -> {
+ if (!baseFileRecordsMap.containsKey(secondaryKey)) {
+ List<HoodieRecord<HoodieMetadataPayload>> recordList = logRecords
+ .values()
+ .stream()
+ .map(record -> (HoodieRecord<HoodieMetadataPayload>) record)
+ .collect(Collectors.toList());
+
+ resultMap.put(secondaryKey, recordList);
+ } else {
+ List<HoodieRecord<HoodieMetadataPayload>> baseFileRecords = baseFileRecordsMap.get(secondaryKey);
+ List<HoodieRecord<HoodieMetadataPayload>> resultRecords = new ArrayList<>();
+
+ baseFileRecords.forEach(prevRecord -> {
+ HoodieMetadataPayload prevPayload = prevRecord.getData();
+ String recordKey = prevPayload.getRecordKeyFromSecondaryIndex();
+
+ if (!logRecords.containsKey(recordKey)) {
+ resultRecords.add(prevRecord);
+ } else {
+ // Merge the records
+ HoodieRecord<HoodieMetadataPayload> newRecord = logRecords.get(recordKey);
+ HoodieMetadataPayload newPayload = newRecord.getData();
+
+ assert recordKey.equals(newPayload.getRecordKeyFromSecondaryIndex());
Review Comment:
sorry i had added for some debugging purpose. Fixed now. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1628207181
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataLogRecordReader.java:
##########
@@ -253,7 +253,7 @@ public HoodieMetadataLogRecordReader build() {
}
private boolean shouldUseMetadataMergedLogRecordScanner() {
- return PARTITION_NAME_SECONDARY_INDEX.equals(partitionName);
+ return partitionName.startsWith(PARTITION_NAME_SECONDARY_INDEX_PREFIX);
Review Comment:
note: this is main fix in the latest commit. The issue was not caught in the testing previously because I was only asserting count of records. I have improved the test. Now, we assert the secondary index records as well as check that file pruning happens. Please see `TestSecondaryIndexPruning`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/SecondaryIndexTestBase.scala:
##########
@@ -62,4 +69,54 @@ class SecondaryIndexTestBase extends HoodieSparkClientTestBase {
cleanupResources()
}
+ def verifyQueryPredicate(hudiOpts: Map[String, String], columnName: String): Unit = {
+ mergedDfList = spark.read.format("hudi").options(hudiOpts).load(basePath).repartition(1).cache() :: mergedDfList
+ val secondaryKey = mergedDfList.last.limit(1).collect().map(row => row.getAs(columnName).toString)
+ val dataFilter = EqualTo(attribute(columnName), Literal(secondaryKey(0)))
+ verifyFilePruning(hudiOpts, dataFilter)
+ }
+
+ private def attribute(partition: String): AttributeReference = {
+ AttributeReference(partition, StringType, nullable = true)()
+ }
+
+
+ private def verifyFilePruning(opts: Map[String, String], dataFilter: Expression): Unit = {
Review Comment:
note: the method here and below just help with verifying file pruning happens correctly i.e. with data skipping enabled, filtered file count < data files in latest snapshot, and with data skipping disabled they should be equal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1630727156
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SecondaryIndexSupport.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.RecordLevelIndexSupport.filterQueryWithRecordKey
+import org.apache.hudi.SecondaryIndexSupport.filterQueriesWithSecondaryKey
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.FileSlice
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.metadata.HoodieTableMetadataUtil.PARTITION_NAME_SECONDARY_INDEX
+import org.apache.hudi.storage.StoragePath
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+import scala.collection.JavaConverters._
+import scala.collection.{JavaConverters, mutable}
+
+class SecondaryIndexSupport(spark: SparkSession,
+ metadataConfig: HoodieMetadataConfig,
+ metaClient: HoodieTableMetaClient) extends RecordLevelIndexSupport(spark, metadataConfig, metaClient) {
+ override def getIndexName: String = SecondaryIndexSupport.INDEX_NAME
+
+ override def computeCandidateFileNames(fileIndex: HoodieFileIndex,
+ queryFilters: Seq[Expression],
+ queryReferencedColumns: Seq[String],
+ prunedPartitionsAndFileSlices: Seq[(Option[BaseHoodieTableFileIndex.PartitionPath], Seq[FileSlice])],
+ shouldPushDownFilesFilter: Boolean
+ ): Option[Set[String]] = {
+ val secondaryKeyConfigOpt = getSecondaryKeyConfig(queryReferencedColumns, metaClient)
+ if (secondaryKeyConfigOpt.isEmpty) {
+ Option.empty
Review Comment:
our scala checkstyle does not allow to add return - https://github.com/apache/hudi/blob/e0cf4b67a4ec4fbf345563dac19170220f306b58/style/scalastyle.xml#L75
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2153048128
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
}, {
"hash" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* bb3bff3f12c7472714f86aad930737238c1a310d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247)
* fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2138822123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124)
* a602c9c4234062e66877fc4bf2c50f94f43767bc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2140514821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a602c9c4234062e66877fc4bf2c50f94f43767bc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142)
* 3c52961bdbcb210e4c7140f5939143cfda7adb50 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2099041393
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e50a635f9dad16057b0c35121716712856a8511 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730)
* 49f3c3fb9f2d802b49b1553db7285e4310faefe4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2099161091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49f3c3fb9f2d802b49b1553db7285e4310faefe4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1630728589
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/SecondaryIndexTestBase.scala:
##########
@@ -62,4 +69,54 @@ class SecondaryIndexTestBase extends HoodieSparkClientTestBase {
cleanupResources()
}
+ def verifyQueryPredicate(hudiOpts: Map[String, String], columnName: String): Unit = {
+ mergedDfList = spark.read.format("hudi").options(hudiOpts).load(basePath).repartition(1).cache() :: mergedDfList
+ val secondaryKey = mergedDfList.last.limit(1).collect().map(row => row.getAs(columnName).toString)
+ val dataFilter = EqualTo(attribute(columnName), Literal(secondaryKey(0)))
+ verifyFilePruning(hudiOpts, dataFilter)
+ }
+
+ private def attribute(partition: String): AttributeReference = {
+ AttributeReference(partition, StringType, nullable = true)()
+ }
+
+
+ private def verifyFilePruning(opts: Map[String, String], dataFilter: Expression): Unit = {
+ // with data skipping
+ val commonOpts = opts + ("path" -> basePath)
+ metaClient = HoodieTableMetaClient.reload(metaClient)
+ var fileIndex = HoodieFileIndex(spark, metaClient, None, commonOpts, includeLogFiles = true)
+ val filteredPartitionDirectories = fileIndex.listFiles(Seq(), Seq(dataFilter))
+ val filteredFilesCount = filteredPartitionDirectories.flatMap(s => s.files).size
+ assertTrue(filteredFilesCount < getLatestDataFilesCount(opts))
Review Comment:
will check this. I also need to refactor and put these in some test util class as it is used in other index test as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2119834319
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afa509c76da8cd4215d962749620fb9896f564ea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002)
* 1c2c253f3316324240407570d8f1e063773bddd7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2119751153
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49f3c3fb9f2d802b49b1553db7285e4310faefe4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746)
* afa509c76da8cd4215d962749620fb9896f564ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2120437419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96b10a6c9a2457512cbe8bb33bb974496aa515c0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137128541
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96b10a6c9a2457512cbe8bb33bb974496aa515c0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006)
* 3d5e22743cf22eba8476470a07876561577b0d8f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137142292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96b10a6c9a2457512cbe8bb33bb974496aa515c0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006)
* 3d5e22743cf22eba8476470a07876561577b0d8f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2097634038
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e50a635f9dad16057b0c35121716712856a8511 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2144549129
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c52961bdbcb210e4c7140f5939143cfda7adb50 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151)
* df12fa59cbba5b14bb98d66dffb510f5b1659177 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195)
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2144576528
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df12fa59cbba5b14bb98d66dffb510f5b1659177 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195)
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* b00831e0a0506714d27bc2a64e58084b357a83cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1624007332
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSecondaryIndexWithSql.scala:
##########
@@ -95,4 +97,39 @@ class TestSecondaryIndexWithSql extends SecondaryIndexTestBase {
private def checkAnswer(sql: String)(expects: Seq[Any]*): Unit = {
assertResult(expects.map(row => Row(row: _*)).toArray.sortBy(_.toString()))(spark.sql(sql).collect().sortBy(_.toString()))
}
+
+ @Test
+ def testSecondaryIndexWithInFilter(): Unit = {
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ var hudiOpts = commonOpts
+ hudiOpts = hudiOpts + (
+ DataSourceWriteOptions.TABLE_TYPE.key -> HoodieTableType.COPY_ON_WRITE.name(),
+ DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true")
+
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | record_key_col string,
+ | not_record_key_col string,
+ | partition_key_col string
+ |) using hudi
+ | options (
+ | primaryKey ='record_key_col',
+ | hoodie.metadata.enable = 'true',
+ | hoodie.metadata.record.index.enable = 'true',
+ | hoodie.datasource.write.recordkey.field = 'record_key_col',
+ | hoodie.enable.data.skipping = 'true'
+ | )
+ | partitioned by(partition_key_col)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values('row1', 'abc', 'p1')")
Review Comment:
do we have test case for non-string values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2144793701
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* b00831e0a0506714d27bc2a64e58084b357a83cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2145735114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* b00831e0a0506714d27bc2a64e58084b357a83cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196)
* ab2875d506fbb642636ca10d044fa9b9e5c951ae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2145850157
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* ab2875d506fbb642636ca10d044fa9b9e5c951ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #11162:
URL: https://github.com/apache/hudi/pull/11162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1628190224
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSecondaryIndexWithSql.scala:
##########
@@ -95,4 +97,39 @@ class TestSecondaryIndexWithSql extends SecondaryIndexTestBase {
private def checkAnswer(sql: String)(expects: Seq[Any]*): Unit = {
assertResult(expects.map(row => Row(row: _*)).toArray.sortBy(_.toString()))(spark.sql(sql).collect().sortBy(_.toString()))
}
+
+ @Test
+ def testSecondaryIndexWithInFilter(): Unit = {
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ var hudiOpts = commonOpts
+ hudiOpts = hudiOpts + (
+ DataSourceWriteOptions.TABLE_TYPE.key -> HoodieTableType.COPY_ON_WRITE.name(),
+ DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true")
+
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | record_key_col string,
+ | not_record_key_col string,
+ | partition_key_col string
+ |) using hudi
+ | options (
+ | primaryKey ='record_key_col',
+ | hoodie.metadata.enable = 'true',
+ | hoodie.metadata.record.index.enable = 'true',
+ | hoodie.datasource.write.recordkey.field = 'record_key_col',
+ | hoodie.enable.data.skipping = 'true'
+ | )
+ | partitioned by(partition_key_col)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values('row1', 'abc', 'p1')")
Review Comment:
Fixed the issue in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1628193853
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -229,6 +231,13 @@ public void cancelAllJobs() {
javaSparkContext.cancelAllJobs();
}
+ @Override
+ public <I, O> O aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O, I, O> seqOp, Functions.Function2<O, O, O> combOp) {
+ Function2<O, I, O> seqOpFunc = seqOp::apply;
+ Function2<O, O, O> combOpFunc = combOp::apply;
+ return HoodieJavaRDD.getJavaRDD(data).aggregate(zeroValue, seqOpFunc, combOpFunc);
Review Comment:
Please check latest commit. I have done the changes and added a test where we create secondary index for a field of long type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2154087365
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
}, {
"hash" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249",
"triggerID" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"triggerType" : "PUSH"
}, {
"hash" : "db219f61a2efd1b601ba1ec14f7e2871df5d615d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24265",
"triggerID" : "db219f61a2efd1b601ba1ec14f7e2871df5d615d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249)
* db219f61a2efd1b601ba1ec14f7e2871df5d615d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24265)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2119810460
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afa509c76da8cd4215d962749620fb9896f564ea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2119741689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49f3c3fb9f2d802b49b1553db7285e4310faefe4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746)
* afa509c76da8cd4215d962749620fb9896f564ea UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2152695612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* 19b6362eb1d0e7c2ea15465964bef441b441106f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244)
* bb3bff3f12c7472714f86aad930737238c1a310d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2152676340
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* 19b6362eb1d0e7c2ea15465964bef441b441106f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244)
* bb3bff3f12c7472714f86aad930737238c1a310d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1629990583
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/HoodieSparkFunctionalIndexClient.java:
##########
@@ -85,6 +85,7 @@ public void create(HoodieTableMetaClient metaClient, String indexName, String in
if (indexExists(metaClient, indexName)) {
throw new HoodieFunctionalIndexException("Index already exists: " + indexName);
}
+ checkArgument(columns.size() == 1, "Only one column can be indexed for functional or secondary index.");
Review Comment:
note: HUDI-7836 to track the support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137216421
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d5e22743cf22eba8476470a07876561577b0d8f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2138810306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124)
* a602c9c4234062e66877fc4bf2c50f94f43767bc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137874427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cdba2db5a4a61458263e31b7306c91ba7c79196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121)
* 9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2139067855
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a602c9c4234062e66877fc4bf2c50f94f43767bc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2119929616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c2c253f3316324240407570d8f1e063773bddd7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2140765294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c52961bdbcb210e4c7140f5939143cfda7adb50 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137503251
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d5e22743cf22eba8476470a07876561577b0d8f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119)
* 1cdba2db5a4a61458263e31b7306c91ba7c79196 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2137858298
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cdba2db5a4a61458263e31b7306c91ba7c79196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121)
* 9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2144454270
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c52961bdbcb210e4c7140f5939143cfda7adb50 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151)
* df12fa59cbba5b14bb98d66dffb510f5b1659177 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1624767426
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSecondaryIndexWithSql.scala:
##########
@@ -95,4 +97,39 @@ class TestSecondaryIndexWithSql extends SecondaryIndexTestBase {
private def checkAnswer(sql: String)(expects: Seq[Any]*): Unit = {
assertResult(expects.map(row => Row(row: _*)).toArray.sortBy(_.toString()))(spark.sql(sql).collect().sortBy(_.toString()))
}
+
+ @Test
+ def testSecondaryIndexWithInFilter(): Unit = {
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ var hudiOpts = commonOpts
+ hudiOpts = hudiOpts + (
+ DataSourceWriteOptions.TABLE_TYPE.key -> HoodieTableType.COPY_ON_WRITE.name(),
+ DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true")
+
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | record_key_col string,
+ | not_record_key_col string,
+ | partition_key_col string
+ |) using hudi
+ | options (
+ | primaryKey ='record_key_col',
+ | hoodie.metadata.enable = 'true',
+ | hoodie.metadata.record.index.enable = 'true',
+ | hoodie.datasource.write.recordkey.field = 'record_key_col',
+ | hoodie.enable.data.skipping = 'true'
+ | )
+ | partitioned by(partition_key_col)
+ | location '$basePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values('row1', 'abc', 'p1')")
Review Comment:
i've added now.. but i discovered one issue while testing which I am still fixing.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SecondaryIndexSupport.scala:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.RecordLevelIndexSupport.filterQueryWithRecordKey
+import org.apache.hudi.SecondaryIndexSupport.filterQueriesWithSecondaryKey
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.FileSlice
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.metadata.HoodieTableMetadataUtil.PARTITION_NAME_SECONDARY_INDEX
+import org.apache.hudi.storage.StoragePath
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+import scala.collection.JavaConverters._
+import scala.collection.{JavaConverters, mutable}
+
+class SecondaryIndexSupport(spark: SparkSession,
+ metadataConfig: HoodieMetadataConfig,
+ metaClient: HoodieTableMetaClient) extends RecordLevelIndexSupport(spark, metadataConfig, metaClient) {
+ override def getIndexName: String = SecondaryIndexSupport.INDEX_NAME
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1624771184
##########
hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java:
##########
@@ -312,6 +312,33 @@ public Map<String, List<HoodieRecordGlobalLocation>> readRecordIndex(List<String
return recordKeyToLocation;
}
+ /**
+ * Get record-location using secondary-index and record-index
+ * <p>
+ * If the Metadata Table is not enabled, an exception is thrown to distinguish this from the absence of the key.
+ *
+ * @param secondaryKeys The list of secondary keys to read
+ */
+ @Override
+ public Map<String, List<HoodieRecordGlobalLocation>> readSecondaryIndex(List<String> secondaryKeys) {
+ ValidationUtils.checkState(dataMetaClient.getTableConfig().isMetadataPartitionAvailable(MetadataPartitionType.RECORD_INDEX),
+ "Record index is not initialized in MDT");
+ ValidationUtils.checkState(
+ dataMetaClient.getTableConfig().getMetadataPartitions().stream().anyMatch(partitionName -> partitionName.startsWith(MetadataPartitionType.SECONDARY_INDEX.getPartitionPath())),
+ "Secondary index is not initialized in MDT");
+ // Fetch secondary-index records
+ Map<String, List<HoodieRecord<HoodieMetadataPayload>>> secondaryKeyRecords = getSecondaryIndexRecords(secondaryKeys, MetadataPartitionType.SECONDARY_INDEX.getPartitionPath());
+ // Now collect the record-keys and fetch the RLI records
Review Comment:
No, here it is RLI. Secondary index contains mapping from secondary key to primary key. So, we need to lookup RLI to get the files for those matching primary keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2096646442
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7952ad87456d56a737eb43813807d64551ddc02b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2151826829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* 19b6362eb1d0e7c2ea15465964bef441b441106f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11162:
URL: https://github.com/apache/hudi/pull/11162#issuecomment-2153061229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7952ad87456d56a737eb43813807d64551ddc02b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23712",
"triggerID" : "7952ad87456d56a737eb43813807d64551ddc02b",
"triggerType" : "PUSH"
}, {
"hash" : "8e50a635f9dad16057b0c35121716712856a8511",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23730",
"triggerID" : "8e50a635f9dad16057b0c35121716712856a8511",
"triggerType" : "PUSH"
}, {
"hash" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23746",
"triggerID" : "49f3c3fb9f2d802b49b1553db7285e4310faefe4",
"triggerType" : "PUSH"
}, {
"hash" : "afa509c76da8cd4215d962749620fb9896f564ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24002",
"triggerID" : "afa509c76da8cd4215d962749620fb9896f564ea",
"triggerType" : "PUSH"
}, {
"hash" : "1c2c253f3316324240407570d8f1e063773bddd7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24003",
"triggerID" : "1c2c253f3316324240407570d8f1e063773bddd7",
"triggerType" : "PUSH"
}, {
"hash" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24006",
"triggerID" : "96b10a6c9a2457512cbe8bb33bb974496aa515c0",
"triggerType" : "PUSH"
}, {
"hash" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24119",
"triggerID" : "3d5e22743cf22eba8476470a07876561577b0d8f",
"triggerType" : "PUSH"
}, {
"hash" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24121",
"triggerID" : "1cdba2db5a4a61458263e31b7306c91ba7c79196",
"triggerType" : "PUSH"
}, {
"hash" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24124",
"triggerID" : "9d0e80222f6cc69b2dba6f4cdbfc642f31a95e52",
"triggerType" : "PUSH"
}, {
"hash" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24142",
"triggerID" : "a602c9c4234062e66877fc4bf2c50f94f43767bc",
"triggerType" : "PUSH"
}, {
"hash" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24151",
"triggerID" : "3c52961bdbcb210e4c7140f5939143cfda7adb50",
"triggerType" : "PUSH"
}, {
"hash" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24195",
"triggerID" : "df12fa59cbba5b14bb98d66dffb510f5b1659177",
"triggerType" : "PUSH"
}, {
"hash" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b342d8f8e10f77419bf1bd0bc9f626a596ad65f9",
"triggerType" : "PUSH"
}, {
"hash" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24196",
"triggerID" : "b00831e0a0506714d27bc2a64e58084b357a83cc",
"triggerType" : "PUSH"
}, {
"hash" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24203",
"triggerID" : "ab2875d506fbb642636ca10d044fa9b9e5c951ae",
"triggerType" : "PUSH"
}, {
"hash" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24217",
"triggerID" : "c6d07ea56ebf1c7eaeb9306df8fe0dd366d72abe",
"triggerType" : "PUSH"
}, {
"hash" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24232",
"triggerID" : "7a0a21f67d6cfc5a17cd1e04abec99dfb6fd53f1",
"triggerType" : "PUSH"
}, {
"hash" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24233",
"triggerID" : "8a9986ae4b8712c0e2e700aeb40a1e4c041fde0e",
"triggerType" : "PUSH"
}, {
"hash" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24244",
"triggerID" : "19b6362eb1d0e7c2ea15465964bef441b441106f",
"triggerType" : "PUSH"
}, {
"hash" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247",
"triggerID" : "bb3bff3f12c7472714f86aad930737238c1a310d",
"triggerType" : "PUSH"
}, {
"hash" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249",
"triggerID" : "fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b342d8f8e10f77419bf1bd0bc9f626a596ad65f9 UNKNOWN
* bb3bff3f12c7472714f86aad930737238c1a310d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24247)
* fe352bc8ce928773ff2c997c58b3ae1c07ae9ba9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1630728024
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SecondaryIndexSupport.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.RecordLevelIndexSupport.filterQueryWithRecordKey
+import org.apache.hudi.SecondaryIndexSupport.filterQueriesWithSecondaryKey
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.FileSlice
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.metadata.HoodieTableMetadataUtil.PARTITION_NAME_SECONDARY_INDEX
+import org.apache.hudi.storage.StoragePath
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+import scala.collection.JavaConverters._
+import scala.collection.{JavaConverters, mutable}
+
+class SecondaryIndexSupport(spark: SparkSession,
+ metadataConfig: HoodieMetadataConfig,
+ metaClient: HoodieTableMetaClient) extends RecordLevelIndexSupport(spark, metadataConfig, metaClient) {
+ override def getIndexName: String = SecondaryIndexSupport.INDEX_NAME
+
+ override def computeCandidateFileNames(fileIndex: HoodieFileIndex,
+ queryFilters: Seq[Expression],
+ queryReferencedColumns: Seq[String],
+ prunedPartitionsAndFileSlices: Seq[(Option[BaseHoodieTableFileIndex.PartitionPath], Seq[FileSlice])],
+ shouldPushDownFilesFilter: Boolean
+ ): Option[Set[String]] = {
+ val secondaryKeyConfigOpt = getSecondaryKeyConfig(queryReferencedColumns, metaClient)
+ if (secondaryKeyConfigOpt.isEmpty) {
+ Option.empty
+ }
+ lazy val (_, secondaryKeys) = if (isIndexAvailable) filterQueriesWithSecondaryKey(queryFilters, secondaryKeyConfigOpt.map(_._2)) else (List.empty, List.empty)
+ if (isIndexAvailable && queryFilters.nonEmpty && secondaryKeys.nonEmpty) {
+ val allFiles = fileIndex.inputFiles.map(strPath => new StoragePath(strPath)).toSeq
Review Comment:
yes, let me do it in a followup PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7146] Integrate secondary index on reader path [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1630734953
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SecondaryIndexSupport.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.RecordLevelIndexSupport.filterQueryWithRecordKey
+import org.apache.hudi.SecondaryIndexSupport.filterQueriesWithSecondaryKey
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.FileSlice
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.metadata.HoodieTableMetadataUtil.PARTITION_NAME_SECONDARY_INDEX
+import org.apache.hudi.storage.StoragePath
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+import scala.collection.JavaConverters._
+import scala.collection.{JavaConverters, mutable}
+
+class SecondaryIndexSupport(spark: SparkSession,
+ metadataConfig: HoodieMetadataConfig,
+ metaClient: HoodieTableMetaClient) extends RecordLevelIndexSupport(spark, metadataConfig, metaClient) {
+ override def getIndexName: String = SecondaryIndexSupport.INDEX_NAME
+
+ override def computeCandidateFileNames(fileIndex: HoodieFileIndex,
+ queryFilters: Seq[Expression],
+ queryReferencedColumns: Seq[String],
+ prunedPartitionsAndFileSlices: Seq[(Option[BaseHoodieTableFileIndex.PartitionPath], Seq[FileSlice])],
+ shouldPushDownFilesFilter: Boolean
+ ): Option[Set[String]] = {
+ val secondaryKeyConfigOpt = getSecondaryKeyConfig(queryReferencedColumns, metaClient)
+ if (secondaryKeyConfigOpt.isEmpty) {
+ Option.empty
Review Comment:
oh, got it. But this judgment logic has no effect here when secondaryKey is empty, will continue to execute the latter code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org