You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/30 02:50:18 UTC
[GitHub] [hudi] nsivabalan opened a new pull request, #7337: [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336)
nsivabalan opened a new pull request, #7337:
URL: https://github.com/apache/hudi/pull/7337
### Change Logs
Optimizing the initial commit w/ metadata table by leveraging bulk_insert instead of upsert.
### Impact
Initial commit w/ metadata table will see much better perf after this patch.
### Risk level (write none, low medium or high below)
Medium.
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336) [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #7337: [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336)
URL: https://github.com/apache/hudi/pull/7337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7337: [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336)
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7337:
URL: https://github.com/apache/hudi/pull/7337#issuecomment-1331797401
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13338",
"triggerID" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"triggerType" : "PUSH"
}, {
"hash" : "e9cf10822ea57aacd9ba0741a85a2a60a9538034",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13342",
"triggerID" : "e9cf10822ea57aacd9ba0741a85a2a60a9538034",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e9cf10822ea57aacd9ba0741a85a2a60a9538034 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13342)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7337: [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336)
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7337:
URL: https://github.com/apache/hudi/pull/7337#issuecomment-1331631327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13338",
"triggerID" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"triggerType" : "PUSH"
}, {
"hash" : "e9cf10822ea57aacd9ba0741a85a2a60a9538034",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13342",
"triggerID" : "e9cf10822ea57aacd9ba0741a85a2a60a9538034",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13338)
* e9cf10822ea57aacd9ba0741a85a2a60a9538034 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13342)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7337: [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336)
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #7337:
URL: https://github.com/apache/hudi/pull/7337#discussion_r1035479716
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -711,15 +713,16 @@ private void initializeFileGroups(HoodieTableMetaClient dataMetaClient, Metadata
LOG.info(String.format("Creating %d file groups for partition %s with base fileId %s at instant time %s",
fileGroupCount, metadataPartition.getPartitionPath(), metadataPartition.getFileIdPrefix(), instantTime));
for (int i = 0; i < fileGroupCount; ++i) {
- final String fileGroupFileId = String.format("%s%04d", metadataPartition.getFileIdPrefix(), i);
+ // Suffix of -0 is used as the file-index (see HoodieCreateHandle) as we want a single file per fileGroup
Review Comment:
Note to reviewer: Please note the change in fileId format.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:
##########
@@ -162,8 +170,10 @@ protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<
// clean plan is the same, so we don't need to delete the requested and inflight instant
// files in the active timeline.
}
-
- List<WriteStatus> statuses = writeClient.upsertPreppedRecords(preppedRecordRDD, instantTime).collect();
+
+ List<WriteStatus> statuses = partitioner.isPresent()
Review Comment:
we are using partitioner to deduce whether its first commit or subsequent one.
wondering if we can add a new commitMethod altogether only for initial commit and keep it light weight (we don't need to trigger compaction, cleaning etc) and directly do bulk_insert for eg.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -258,6 +268,30 @@ public static boolean metadataPartitionExists(String basePath, HoodieEngineConte
}
}
+ /**
+ * Tag each record with the location in the given partition.
+ *
+ * The record is tagged with respective file slice's location based on its record key.
+ */
+ public static HoodieData<HoodieRecord> tagRecordsWithLocation(HoodieTableMetaClient metadataMetaClient, HoodieData<HoodieRecord> records, String partitionName) {
Review Comment:
Note to reviewer: tagging the records upfront here w/ actual location
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -189,24 +187,28 @@ private void enablePartitions() {
final HoodieMetadataConfig metadataConfig = dataWriteConfig.getMetadataConfig();
boolean isBootstrapCompleted;
Option<HoodieTableMetaClient> metaClient = Option.empty();
- try {
- isBootstrapCompleted = dataMetaClient.getFs().exists(new Path(metadataWriteConfig.getBasePath(), HoodieTableMetaClient.METAFOLDER_NAME));
- if (isBootstrapCompleted) {
- metaClient = Option.of(HoodieTableMetaClient.builder().setConf(hadoopConf.get())
- .setBasePath(metadataWriteConfig.getBasePath()).build());
- }
- } catch (IOException e) {
- throw new HoodieException("Failed to enable metadata partitions!", e);
+ // TODO: check for metadata/hoodie.properties instead of just metadata dir.
+ isBootstrapCompleted = dataMetaClient.getTableConfig().isMetadataTableEnabled();
+ if (isBootstrapCompleted) {
+ metaClient = Option.of(HoodieTableMetaClient.builder().setConf(hadoopConf.get())
+ .setBasePath(metadataWriteConfig.getBasePath()).build());
}
Option<HoodieTableFileSystemView> fsView = Option.ofNullable(
metaClient.isPresent() ? HoodieTableMetadataUtil.getFileSystemView(metaClient.get()) : null);
enablePartition(MetadataPartitionType.FILES, metadataConfig, metaClient, fsView, isBootstrapCompleted);
- if (metadataConfig.isBloomFilterIndexEnabled()) {
- enablePartition(MetadataPartitionType.BLOOM_FILTERS, metadataConfig, metaClient, fsView, isBootstrapCompleted);
- }
- if (metadataConfig.isColumnStatsIndexEnabled()) {
- enablePartition(MetadataPartitionType.COLUMN_STATS, metadataConfig, metaClient, fsView, isBootstrapCompleted);
+ // to initialize new metadata partitions, since we use bulk_insert prepped, we can only initialize one new metadata partition at a time.
Review Comment:
Note to reviewer: Need to consult w/ prashanth as to why we have this new requirement (i.e. initializing only one partition at a time w/ MDT)
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -110,6 +113,13 @@ public class HoodieTableMetadataUtil {
public static final String PARTITION_NAME_COLUMN_STATS = "column_stats";
public static final String PARTITION_NAME_BLOOM_FILTERS = "bloom_filters";
+ // Suffix to use for bootstrapping additional indexes. Should be less than other suffixes.
+ private static final String INDEX_BOOTSTRAP_TIMESTAMP_SUFFIX = "001";
+ // Suffix to use for compaction
+ private static final String COMPACTION_TIMESTAMP_SUFFIX = "002";
Review Comment:
need to consult w/ Prashant here. as per master, compaction has suffix of "001", but w/ this patch we are switching to "002".
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -967,35 +988,39 @@ protected HoodieData<HoodieRecord> prepRecords(Map<MetadataPartitionType,
HoodieData<HoodieRecord>> partitionRecordsMap) {
// The result set
HoodieData<HoodieRecord> allPartitionRecords = engineContext.emptyHoodieData();
-
- HoodieTableFileSystemView fsView = HoodieTableMetadataUtil.getFileSystemView(metadataMetaClient);
Review Comment:
Note to reviewer: since the records are tagged upfront, we don't need these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7337: [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336)
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7337:
URL: https://github.com/apache/hudi/pull/7337#issuecomment-1331584935
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7337: [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336)
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7337:
URL: https://github.com/apache/hudi/pull/7337#issuecomment-1331588917
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13338",
"triggerID" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13338)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7337: [HUDI-5300] Optimizing initial commit w/ metadata table (Stacked on top of pull/7336)
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7337:
URL: https://github.com/apache/hudi/pull/7337#issuecomment-1331628455
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13338",
"triggerID" : "0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45",
"triggerType" : "PUSH"
}, {
"hash" : "e9cf10822ea57aacd9ba0741a85a2a60a9538034",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e9cf10822ea57aacd9ba0741a85a2a60a9538034",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ed5a2bc61e9dff8e5b60e59d2962081d9e4bc45 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13338)
* e9cf10822ea57aacd9ba0741a85a2a60a9538034 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org