You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/18 05:24:50 UTC
[GitHub] [hudi] yihua opened a new pull request, #6432: [WIP][HUDI-4586] Improve metadata fetching in bloom index
yihua opened a new pull request, #6432:
URL: https://github.com/apache/hudi/pull/6432
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1223253241
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
}, {
"hash" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855",
"triggerID" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"triggerType" : "PUSH"
}, {
"hash" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10887",
"triggerID" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab274c8aa0e71723486655e0cedcbdea1ebc3243 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855)
* ed15f57dc58b2e9142dd33a0ecd078bf4c236afc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10887)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1224932396
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10887",
"triggerID" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed15f57dc58b2e9142dd33a0ecd078bf4c236afc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10887)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221372313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0de1341db083aed17caf4364e45c543ec5e20e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221499422
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
}, {
"hash" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855",
"triggerID" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab274c8aa0e71723486655e0cedcbdea1ebc3243 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221477861
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
}, {
"hash" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855",
"triggerID" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51628385c214d692a7a0923c63a82810eaffb8ac Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854)
* ab274c8aa0e71723486655e0cedcbdea1ebc3243 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221476759
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51628385c214d692a7a0923c63a82810eaffb8ac Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua closed pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
URL: https://github.com/apache/hudi/pull/6432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221455787
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0de1341db083aed17caf4364e45c543ec5e20e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851)
* 730e59884e03eaa460d5561851a51b5f1c6d2694 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221469230
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 730e59884e03eaa460d5561851a51b5f1c6d2694 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853)
* 51628385c214d692a7a0923c63a82810eaffb8ac Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6432:
URL: https://github.com/apache/hudi/pull/6432#discussion_r951958281
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/BloomIndexFileInfo.java:
##########
@@ -27,19 +27,20 @@
public class BloomIndexFileInfo implements Serializable {
private final String fileId;
-
+ private final String filename;
Review Comment:
Here we store the filename to avoid recomputation of the same piece of information across stages.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBloomIndex.java:
##########
@@ -121,12 +121,14 @@ private HoodiePairData<HoodieKey, HoodieRecordLocation> lookupIndex(
// Step 2: Load all involved files as <Partition, filename> pairs
List<Pair<String, BloomIndexFileInfo>> fileInfoList = getBloomIndexFileInfoForPartitions(context, hoodieTable, affectedPartitionPathList);
- final Map<String, List<BloomIndexFileInfo>> partitionToFileInfo =
- fileInfoList.stream().collect(groupingBy(Pair::getLeft, mapping(Pair::getRight, toList())));
+ // partition -> {file ID -> BloomIndexFileInfo instance}
+ final Map<String, Map<String, BloomIndexFileInfo>> partitionToFileInfo = fileInfoList.stream()
Review Comment:
Switching from List to Map for each value so that the filename can be retrieved with file ID in O(1) time.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/bloom/SparkHoodieBloomIndexHelper.java:
##########
@@ -82,25 +87,33 @@ public HoodiePairData<HoodieKey, HoodieRecordLocation> findMatchingFilesForRecor
if (config.getBloomIndexUseMetadata()
&& hoodieTable.getMetaClient().getTableConfig().getMetadataPartitions()
.contains(BLOOM_FILTERS.getPartitionPath())) {
- // Step 1: Sort by file id
- JavaRDD<Tuple2<String, HoodieKey>> sortedFileIdAndKeyPairs =
- fileComparisonsRDD.sortBy(Tuple2::_1, true, joinParallelism);
+ // Step 1: Sort by bloom filter key (Hash ID of partition and file name) in metadata table
+ JavaRDD<Tuple2<Tuple2<String, String>, HoodieKey>> sortedFileIdAndKeyPairs =
+ fileComparisonsRDD
+ .sortBy(t -> HoodieMetadataPayload.getBloomFilterIndexKey(
+ new PartitionIndexID(t._1._1), new FileIndexID(t._1._2)), true, joinParallelism);
// Step 2: Use bloom filter to filter and the actual log file to get the record location
- keyLookupResultRDD = sortedFileIdAndKeyPairs.mapPartitionsWithIndex(
- new HoodieMetadataBloomIndexCheckFunction(hoodieTable), true);
+ keyLookupResultRDD = sortedFileIdAndKeyPairs.coalesce(config.getBloomIndexMetadataReadParallelism())
Review Comment:
This limits the parallelism of bloom filter fetching.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBloomIndex.java:
##########
@@ -121,12 +121,14 @@ private HoodiePairData<HoodieKey, HoodieRecordLocation> lookupIndex(
// Step 2: Load all involved files as <Partition, filename> pairs
List<Pair<String, BloomIndexFileInfo>> fileInfoList = getBloomIndexFileInfoForPartitions(context, hoodieTable, affectedPartitionPathList);
- final Map<String, List<BloomIndexFileInfo>> partitionToFileInfo =
- fileInfoList.stream().collect(groupingBy(Pair::getLeft, mapping(Pair::getRight, toList())));
+ // partition -> {file ID -> BloomIndexFileInfo instance}
+ final Map<String, Map<String, BloomIndexFileInfo>> partitionToFileInfo = fileInfoList.stream()
+ .collect(groupingBy(Pair::getLeft, toMap(entry -> entry.getRight().getFileId(), Pair::getRight)));
// Step 3: Obtain a HoodieData, for each incoming record, that already exists, with the file id,
// that contains it.
- HoodieData<Pair<String, HoodieKey>> fileComparisonPairs =
+ // Each entry: ((File ID, Filename), HoodieKey instance)
+ HoodieData<Pair<Pair<String, String>, HoodieKey>> fileComparisonPairs =
Review Comment:
Along the DAG, passing down the file names so they don't need to be recomputed.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBloomIndex.java:
##########
@@ -202,41 +219,57 @@ private List<Pair<String, BloomIndexFileInfo>> getFileInfoForLatestBaseFiles(
* @param partitions - List of partitions for which column stats need to be loaded
* @param context - Engine context
* @param hoodieTable - Hoodie table
+ * @param parallelism - Parallelism for reading column stats from metadata table
* @return List of partition and file column range info pairs
*/
protected List<Pair<String, BloomIndexFileInfo>> loadColumnRangesFromMetaIndex(
- List<String> partitions, final HoodieEngineContext context, final HoodieTable<?, ?, ?, ?> hoodieTable) {
+ List<String> partitions, final HoodieEngineContext context,
+ final HoodieTable<?, ?, ?, ?> hoodieTable, int parallelism) {
// also obtain file ranges, if range pruning is enabled
context.setJobStatus(this.getClass().getName(), "Load meta index key ranges for file slices: " + config.getTableName());
final String keyField = hoodieTable.getMetaClient().getTableConfig().getRecordKeyFieldProp();
- return context.flatMap(partitions, partitionName -> {
- // Partition and file name pairs
- List<Pair<String, String>> partitionFileNameList = HoodieIndexUtils.getLatestBaseFilesForPartition(partitionName,
- hoodieTable).stream().map(baseFile -> Pair.of(partitionName, baseFile.getFileName()))
- .sorted()
- .collect(toList());
- if (partitionFileNameList.isEmpty()) {
- return Stream.empty();
- }
- try {
- Map<Pair<String, String>, HoodieMetadataColumnStats> fileToColumnStatsMap =
- hoodieTable.getMetadataTable().getColumnStats(partitionFileNameList, keyField);
- List<Pair<String, BloomIndexFileInfo>> result = new ArrayList<>();
- for (Map.Entry<Pair<String, String>, HoodieMetadataColumnStats> entry : fileToColumnStatsMap.entrySet()) {
- result.add(Pair.of(entry.getKey().getLeft(),
- new BloomIndexFileInfo(
- FSUtils.getFileId(entry.getKey().getRight()),
- // NOTE: Here we assume that the type of the primary key field is string
- (String) unwrapStatisticValueWrapper(entry.getValue().getMinValue()),
- (String) unwrapStatisticValueWrapper(entry.getValue().getMaxValue())
- )));
- }
- return result.stream();
- } catch (MetadataNotFoundException me) {
- throw new HoodieMetadataException("Unable to find column range metadata for partition:" + partitionName, me);
- }
- }, Math.max(partitions.size(), 1));
+ return context.parallelize(partitions, Math.max(Math.min(partitions.size(), parallelism), 1))
Review Comment:
This limits the parallelism of column stats fetching from the metadata table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221455322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0de1341db083aed17caf4364e45c543ec5e20e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851)
* 730e59884e03eaa460d5561851a51b5f1c6d2694 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221463047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0de1341db083aed17caf4364e45c543ec5e20e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851)
* 730e59884e03eaa460d5561851a51b5f1c6d2694 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853)
* 51628385c214d692a7a0923c63a82810eaffb8ac Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1421870414
Part of this PR went into #7642 so closing this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1223375990
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
}, {
"hash" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855",
"triggerID" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"triggerType" : "PUSH"
}, {
"hash" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10887",
"triggerID" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed15f57dc58b2e9142dd33a0ecd078bf4c236afc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10887)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221382433
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0de1341db083aed17caf4364e45c543ec5e20e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6432:
URL: https://github.com/apache/hudi/pull/6432#discussion_r954187229
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/BloomIndexFileInfo.java:
##########
@@ -27,19 +27,20 @@
public class BloomIndexFileInfo implements Serializable {
private final String fileId;
-
+ private final String filename;
private final String minRecordKey;
-
private final String maxRecordKey;
- public BloomIndexFileInfo(String fileId, String minRecordKey, String maxRecordKey) {
+ public BloomIndexFileInfo(String fileId, String filename, String minRecordKey, String maxRecordKey) {
this.fileId = fileId;
+ this.filename = filename;
Review Comment:
nit: `fileName`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/BloomIndexFileInfo.java:
##########
@@ -27,19 +27,20 @@
public class BloomIndexFileInfo implements Serializable {
private final String fileId;
-
+ private final String filename;
Review Comment:
Do we really need to store both file-id and file-name? I think we can just store the file-name, and then convert it to file-id wherever necessary
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/bloom/HoodieMetadataBloomIndexCheckFunction.java:
##########
@@ -83,37 +89,64 @@ protected void start() {
@Override
protected List<HoodieKeyLookupResult> computeNext() {
// Partition path and file name pair to list of keys
- final Map<Pair<String, String>, List<HoodieKey>> fileToKeysMap = new HashMap<>();
- final Map<String, HoodieBaseFile> fileIDBaseFileMap = new HashMap<>();
+ final Map<Pair<String, String>, List<HoodieKey>> batchFileToKeysMap = new HashMap<>();
final List<HoodieKeyLookupResult> resultList = new ArrayList<>();
+ String lastFileId = null;
+
+ try {
+ // Here we batch process the lookup of bloom filters in metadata table
+ // assuming the partition path and file name pairs are already sorted by the corresponding key
+ while (inputItr.hasNext()) {
+ Tuple2<Tuple2<String, String>, HoodieKey> entry = inputItr.next();
+ final String partitionPath = entry._2.getPartitionPath();
+ final String fileId = entry._1._1();
+ final String filename = entry._1._2();
+
+ if (lastFileId == null || !lastFileId.equals(fileId)) {
+ if (processedFileIdSet.contains(fileId)) {
+ LOG.warn(String.format("Fetching the bloom filter for file ID %s again. "
+ + " The input pairs of file ID and record key are not sorted.", fileId));
+ }
+ lastFileId = fileId;
+ processedFileIdSet.add(fileId);
+ }
+
+ batchFileToKeysMap.computeIfAbsent(Pair.of(partitionPath, filename), k -> new ArrayList<>()).add(entry._2);
- while (inputItr.hasNext()) {
- Tuple2<String, HoodieKey> entry = inputItr.next();
- final String partitionPath = entry._2.getPartitionPath();
- final String fileId = entry._1;
- if (!fileIDBaseFileMap.containsKey(fileId)) {
- Option<HoodieBaseFile> baseFile = hoodieTable.getBaseFileOnlyView().getLatestBaseFile(partitionPath, fileId);
- if (!baseFile.isPresent()) {
- throw new HoodieIndexException("Failed to find the base file for partition: " + partitionPath
- + ", fileId: " + fileId);
+ if (batchFileToKeysMap.size() == batchSize) {
+ resultList.addAll(lookupKeysInBloomFilters(batchFileToKeysMap));
+ batchFileToKeysMap.clear();
}
- fileIDBaseFileMap.put(fileId, baseFile.get());
}
- fileToKeysMap.computeIfAbsent(Pair.of(partitionPath, fileIDBaseFileMap.get(fileId).getFileName()),
- k -> new ArrayList<>()).add(entry._2);
- if (fileToKeysMap.size() > BLOOM_FILTER_CHECK_MAX_FILE_COUNT_PER_BATCH) {
- break;
+
+ if (batchFileToKeysMap.size() > 0) {
+ resultList.addAll(lookupKeysInBloomFilters(batchFileToKeysMap));
+ batchFileToKeysMap.clear();
}
+
+ return resultList;
+ } catch (Throwable e) {
+ if (e instanceof HoodieException) {
+ throw e;
+ }
+ throw new HoodieIndexException("Error checking bloom filter using metadata table.", e);
}
- if (fileToKeysMap.isEmpty()) {
- return Collections.emptyList();
- }
+ }
+
+ @Override
+ protected void end() {
+ }
- List<Pair<String, String>> partitionNameFileNameList = new ArrayList<>(fileToKeysMap.keySet());
+ private List<HoodieKeyLookupResult> lookupKeysInBloomFilters(
+ Map<Pair<String, String>, List<HoodieKey>> fileToKeysMap) {
+ List<HoodieKeyLookupResult> resultList = new ArrayList<>();
+ List<Pair<String, String>> partitionPathFileNameList = new ArrayList<>(fileToKeysMap.keySet());
+ HoodieTimer timer = HoodieTimer.start();
Map<Pair<String, String>, BloomFilter> fileToBloomFilterMap =
- hoodieTable.getMetadataTable().getBloomFilters(partitionNameFileNameList);
+ hoodieTable.getMetadataTable().getBloomFilters(partitionPathFileNameList);
+ LOG.error(String.format("Took %d ms to look up %s bloom filters",
Review Comment:
This should rather be debug/info
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/bloom/HoodieMetadataBloomIndexCheckFunction.java:
##########
@@ -53,26 +55,30 @@
* keys are checked against them.
*/
public class HoodieMetadataBloomIndexCheckFunction implements
- Function2<Integer, Iterator<Tuple2<String, HoodieKey>>, Iterator<List<HoodieKeyLookupResult>>> {
+ Function2<Integer, Iterator<Tuple2<Tuple2<String, String>, HoodieKey>>, Iterator<List<HoodieKeyLookupResult>>> {
Review Comment:
In general, it's better to keep the nested hierarchies as flat as possible (makes it easier to comprehend, reduces amount of objects needed as well): in that case we could use just the `Tuple3` in lieu of 2 `Tuple2`s
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBloomIndex.java:
##########
@@ -121,12 +121,14 @@ private HoodiePairData<HoodieKey, HoodieRecordLocation> lookupIndex(
// Step 2: Load all involved files as <Partition, filename> pairs
List<Pair<String, BloomIndexFileInfo>> fileInfoList = getBloomIndexFileInfoForPartitions(context, hoodieTable, affectedPartitionPathList);
- final Map<String, List<BloomIndexFileInfo>> partitionToFileInfo =
- fileInfoList.stream().collect(groupingBy(Pair::getLeft, mapping(Pair::getRight, toList())));
+ // partition -> {file ID -> BloomIndexFileInfo instance}
+ final Map<String, Map<String, BloomIndexFileInfo>> partitionToFileInfo = fileInfoList.stream()
+ .collect(groupingBy(Pair::getLeft, toMap(entry -> entry.getRight().getFileId(), Pair::getRight)));
// Step 3: Obtain a HoodieData, for each incoming record, that already exists, with the file id,
// that contains it.
- HoodieData<Pair<String, HoodieKey>> fileComparisonPairs =
+ // Each entry: ((File ID, Filename), HoodieKey instance)
+ HoodieData<Pair<Pair<String, String>, HoodieKey>> fileComparisonPairs =
Review Comment:
Same applies here: i don't think we need to propagate both file-id and file-name since the latter contains the former (it increases both complexity and amount of data we need to shuffle)
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/bloom/HoodieMetadataBloomIndexCheckFunction.java:
##########
@@ -83,37 +89,64 @@ protected void start() {
@Override
protected List<HoodieKeyLookupResult> computeNext() {
// Partition path and file name pair to list of keys
- final Map<Pair<String, String>, List<HoodieKey>> fileToKeysMap = new HashMap<>();
- final Map<String, HoodieBaseFile> fileIDBaseFileMap = new HashMap<>();
+ final Map<Pair<String, String>, List<HoodieKey>> batchFileToKeysMap = new HashMap<>();
final List<HoodieKeyLookupResult> resultList = new ArrayList<>();
+ String lastFileId = null;
+
+ try {
+ // Here we batch process the lookup of bloom filters in metadata table
+ // assuming the partition path and file name pairs are already sorted by the corresponding key
+ while (inputItr.hasNext()) {
+ Tuple2<Tuple2<String, String>, HoodieKey> entry = inputItr.next();
+ final String partitionPath = entry._2.getPartitionPath();
+ final String fileId = entry._1._1();
+ final String filename = entry._1._2();
+
+ if (lastFileId == null || !lastFileId.equals(fileId)) {
+ if (processedFileIdSet.contains(fileId)) {
+ LOG.warn(String.format("Fetching the bloom filter for file ID %s again. "
+ + " The input pairs of file ID and record key are not sorted.", fileId));
+ }
+ lastFileId = fileId;
+ processedFileIdSet.add(fileId);
+ }
+
+ batchFileToKeysMap.computeIfAbsent(Pair.of(partitionPath, filename), k -> new ArrayList<>()).add(entry._2);
- while (inputItr.hasNext()) {
- Tuple2<String, HoodieKey> entry = inputItr.next();
- final String partitionPath = entry._2.getPartitionPath();
- final String fileId = entry._1;
- if (!fileIDBaseFileMap.containsKey(fileId)) {
- Option<HoodieBaseFile> baseFile = hoodieTable.getBaseFileOnlyView().getLatestBaseFile(partitionPath, fileId);
- if (!baseFile.isPresent()) {
- throw new HoodieIndexException("Failed to find the base file for partition: " + partitionPath
- + ", fileId: " + fileId);
+ if (batchFileToKeysMap.size() == batchSize) {
+ resultList.addAll(lookupKeysInBloomFilters(batchFileToKeysMap));
Review Comment:
Let's just break in this conditional to avoid duplication
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBloomIndex.java:
##########
@@ -121,12 +121,14 @@ private HoodiePairData<HoodieKey, HoodieRecordLocation> lookupIndex(
// Step 2: Load all involved files as <Partition, filename> pairs
List<Pair<String, BloomIndexFileInfo>> fileInfoList = getBloomIndexFileInfoForPartitions(context, hoodieTable, affectedPartitionPathList);
- final Map<String, List<BloomIndexFileInfo>> partitionToFileInfo =
- fileInfoList.stream().collect(groupingBy(Pair::getLeft, mapping(Pair::getRight, toList())));
+ // partition -> {file ID -> BloomIndexFileInfo instance}
Review Comment:
nit: "File Id" denomination might be confusing, more accurate would be "file-group id"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1226334636
Following up on my previous comment: taking a deeper look i see following issues in our code at the moment
1. Creating `HoodieTableBackedMetadata` instance w/in `HoodieTable` we don't specify that it should reuse MT readers.
2. `HoodieMetadataMergedLogRecordReader.getRecordsByKeys` always clears previously computed `records` and always scans from scratch while instead we should NOT be re-processing records that have already been processed, and instead just incrementally process missing ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221477338
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
}, {
"hash" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51628385c214d692a7a0923c63a82810eaffb8ac Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854)
* ab274c8aa0e71723486655e0cedcbdea1ebc3243 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221373037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0de1341db083aed17caf4364e45c543ec5e20e1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1223241439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10854",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
}, {
"hash" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855",
"triggerID" : "ab274c8aa0e71723486655e0cedcbdea1ebc3243",
"triggerType" : "PUSH"
}, {
"hash" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab274c8aa0e71723486655e0cedcbdea1ebc3243 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10855)
* ed15f57dc58b2e9142dd33a0ecd078bf4c236afc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1224927464
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed15f57dc58b2e9142dd33a0ecd078bf4c236afc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1224976328
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10887",
"triggerID" : "ed15f57dc58b2e9142dd33a0ecd078bf4c236afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed15f57dc58b2e9142dd33a0ecd078bf4c236afc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10887)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6432: [HUDI-4586] Improve metadata fetching in bloom index
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6432:
URL: https://github.com/apache/hudi/pull/6432#issuecomment-1221456189
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851",
"triggerID" : "f0de1341db083aed17caf4364e45c543ec5e20e1",
"triggerType" : "PUSH"
}, {
"hash" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853",
"triggerID" : "730e59884e03eaa460d5561851a51b5f1c6d2694",
"triggerType" : "PUSH"
}, {
"hash" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51628385c214d692a7a0923c63a82810eaffb8ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0de1341db083aed17caf4364e45c543ec5e20e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10851)
* 730e59884e03eaa460d5561851a51b5f1c6d2694 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10853)
* 51628385c214d692a7a0923c63a82810eaffb8ac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org