You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/19 15:28:18 UTC
[GitHub] [hudi] codope opened a new pull request #4640: [HUDI-3225] Add RFC for async metadata indexing
codope opened a new pull request #4640:
URL: https://github.com/apache/hudi/pull/4640
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1086116842
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
}, {
"hash" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7710",
"triggerID" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ee05ae22e767b1c62862f9d041ef2835d4667f63 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7710)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1065056424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750)
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r812687856
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1065056424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750)
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1063102102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719)
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r824656295
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
Review comment:
i'd prefer `index` for brevity, and none of our action end with -ing. But let me know if you think `indexing` is more appropriate, i can change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r824447974
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,264 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will only write base
+ files but ensure that log entries due to instants after `t` are in the same
+ filegroup i.e. no new filegroup is initialized by writers while indexing is
+ in progress.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes. The indexer will error out if there is no lock provider
+configured.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is pending,
+which means indexer has already intialized a filegroup, then each writer will
+create log files in the same filegroup for the metadata index update. This will
+happen within the existing data table lock.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out.
+Whether indexing timed out or not, table config would be updated with any MDT
+partition(s) for which indexing was complete till `t2`. In case of timeout
+indexer will abort. At this point, user can trigger the index process again,
+however, this time indexer will check for available partitions in table config
+and skip those partitions. This design ensures that the regular writers do not
+fail due to indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. However, table
+config will indicate that partition is not ready to use. When indexer is
+re-triggered, it will check the plan and table config to figure out which MDT
+partitions to index and start indexing for those partitions.
Review comment:
When the indexer starts the next time, it will choose a different instant time. Hence, the older log blocks written are no longer valid. So I think each time the indexer starts (either the first time or after a failure), it should clean out the older file groups and create new ones (with newer instant time).
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,264 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will only write base
Review comment:
will **not** only write base files but **also** ensure
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,264 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will only write base
+ files but ensure that log entries due to instants after `t` are in the same
+ filegroup i.e. no new filegroup is initialized by writers while indexing is
+ in progress.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes. The indexer will error out if there is no lock provider
+configured.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is pending,
+which means indexer has already intialized a filegroup, then each writer will
+create log files in the same filegroup for the metadata index update. This will
+happen within the existing data table lock.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out.
+Whether indexing timed out or not, table config would be updated with any MDT
+partition(s) for which indexing was complete till `t2`. In case of timeout
+indexer will abort. At this point, user can trigger the index process again,
+however, this time indexer will check for available partitions in table config
+and skip those partitions. This design ensures that the regular writers do not
+fail due to indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. However, table
+config will indicate that partition is not ready to use. When indexer is
+re-triggered, it will check the plan and table config to figure out which MDT
+partitions to index and start indexing for those partitions.
+
+**Case 3: Race conditions**
Review comment:
There is another race condition possible:
1. Writer is in inflight mode
2. Indexer is starting and creating the file-groups. Suppose there are 100 file-groups to be created.
3. Writer just finished and tries to write log blocks - it only sees a subset of file-groups created yet (as the above step 2 above has not completed yet). This will cause writer to incorrectly write updated to lesser number of shards.
In essence:
1. Locking is required
2. Indexer need to hold lock which creating the file-groups too
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
Review comment:
Yes.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
Review comment:
+1 for requiring locking.
Having wrong or missing data from the MDT is very difficult to debug in the long run and can have serious data quality issues. Also, anyone having enough scale to be requiring asyc indexing should be able to choose one of the many locking options available.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
Review comment:
Should this be reflected by choosing the index timestamp as t?
E.g. t.index.requested ?
Table service operations on the metadata table usually take in the timestamp of the last op with a suffix - 001 for compaction, 002 for clean etc.
So it may be good to have this as t001. index.requested.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
Review comment:
Metadata table is unique in the respect that each write to MDT will involve multiple partitions to be updated together in a transaction. So I do not see a truly parallel commit to MDT possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r824623608
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,264 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will only write base
+ files but ensure that log entries due to instants after `t` are in the same
+ filegroup i.e. no new filegroup is initialized by writers while indexing is
+ in progress.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes. The indexer will error out if there is no lock provider
+configured.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is pending,
+which means indexer has already intialized a filegroup, then each writer will
+create log files in the same filegroup for the metadata index update. This will
+happen within the existing data table lock.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out.
+Whether indexing timed out or not, table config would be updated with any MDT
+partition(s) for which indexing was complete till `t2`. In case of timeout
+indexer will abort. At this point, user can trigger the index process again,
+however, this time indexer will check for available partitions in table config
+and skip those partitions. This design ensures that the regular writers do not
+fail due to indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. However, table
+config will indicate that partition is not ready to use. When indexer is
+re-triggered, it will check the plan and table config to figure out which MDT
+partitions to index and start indexing for those partitions.
Review comment:
Yes, that's the plan. But, it will start from scratch only for the partitions that were partially indexed i.e. partitions for which table config was not updated in the last indexing. Table config update always happens at the end of indexing for a partition.
We don't want to start all over again for all the partitions. So, let's say at some `t` indexer was scheduled and it wrote `t.index.requested` with plan of indexing `files` and `column_stats` partitions. It completed `files` but failed midway for `column_stats`. Then table config will show that only `files` partition is available for reads/updates. When indexer starts the next time, it will see a pending index action, reads the plan as well as table config and figures out that only `column_stats` index is pending. Will clean the older filegroups for column_stats and choose the latest completed instant (without holes) on data timeline and create new filegroup and so on.
If this sounds right, I can update this example in the RFC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1086042189
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
}, {
"hash" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839)
* ee05ae22e767b1c62862f9d041ef2835d4667f63 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r812688821
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
Review comment:
index is a noun as well as verb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar merged pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #4640:
URL: https://github.com/apache/hudi/pull/4640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1065054306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750)
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1063194983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1062554441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347)
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1062584118
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1065054306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750)
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1063098982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719)
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1063098982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719)
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1086045604
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
}, {
"hash" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7710",
"triggerID" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839)
* ee05ae22e767b1c62862f9d041ef2835d4667f63 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7710)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1086045604
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
}, {
"hash" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7710",
"triggerID" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839)
* ee05ae22e767b1c62862f9d041ef2835d4667f63 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7710)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r825311879
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,281 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned. The creation of filegroups
+ will be done within a lock.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will not only write
+ base files but also ensure that log entries due to instants after `t` are in
+ the same filegroup i.e. no new filegroup is initialized by writers while
+ indexing is in progress.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes. The indexer will error out if there is no lock provider
+configured.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is pending,
+which means indexer has already intialized a filegroup, then each writer will
+create log files in the same filegroup for the metadata index update. This will
+happen within the existing data table lock.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
Review comment:
you're right..i'll word it better..what i meant is run until timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4640: [HUDI-3225] Add RFC for async metadata indexing
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r791273720
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
Review comment:
we should support this via regular writers as well. just scheduling. it will make life easier.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
Review comment:
its not very clear what does the indexer does here apart from populating base files for all instants having commit time < t. for eg, what does indexer does to completed instants > t up until now ?
If I am not wrong, "check for all completed commit actions to ensure each of them added entries per its indexing plan" refers to all instants up until now. would be good to call it out.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
Review comment:
in a single write mode, users may not have configured any lock service and we don't enforce one as of today. something to keep in mind.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
Review comment:
also, lets say this is the timeline when indexer started
C1, C2,.... C5 (inflight), C6, C7, C8. Start indexer.
indexer will build base file with all info until C4.
what happens to data pertaining to C6, C7, C8? these are already completed ones.
essentially indexer will go through every completed commit up until now and ensure they are applied to new partition. if its < C= C4, it goes to base file. if its> C4, it goes as delta commit is it ?
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
Review comment:
I don't see details on when exactly regular writers will start to make synchronous updates. Also, when exactly the callers can start using the new index that got built out? whats the source of truth. we can rely on timeline completed instant for the index, but after archival? also loading timeline everytime might be costly as well.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is inflight and
+a basefile is already being written under the metadata partition, then each
+writer will create log files in the same filegroup for the metadata index
+update. This will happen within the existing data table lock. However, if the
+indexing has still not started and instant `t3` is still in requested state,
+then writer will still continue to log entries but indexer will handle this
+scenario and assign the same filegroup id when `t3` transitions to inflight.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out. After
+timeout, indexer will abort writing the instant upto which indexing was done
+in `t3.index.inflight` file in the timeline. At this point, user can trigger the
+index process again, however, this time `t2` will become the last completed
+instant. This design ensures that the regular writers do not fail due to
+indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. Indexer will
+fetch the last instant for which indexing was done from `.index.inflight` file.
+It will start indexing again from the instant thereafter.
+
+**Case 3: Race conditions**
+
+a) Writer went inflight just after an indexing request was added but indexer has
+not yet started executing.
+
+In this case, writer will continue to log updates in metadata partition. At the
+time of execution, indexer will see there are already some log files and handle
+that in the indexing check.
+
+b) Inflight writer about to commit, but indexing completed just before that.
+
+In this case, since the indexer completed before the writer, so it has already
Review comment:
will sync up directly. I don't get this fully.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is inflight and
+a basefile is already being written under the metadata partition, then each
+writer will create log files in the same filegroup for the metadata index
+update. This will happen within the existing data table lock. However, if the
+indexing has still not started and instant `t3` is still in requested state,
+then writer will still continue to log entries but indexer will handle this
+scenario and assign the same filegroup id when `t3` transitions to inflight.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out. After
+timeout, indexer will abort writing the instant upto which indexing was done
+in `t3.index.inflight` file in the timeline. At this point, user can trigger the
+index process again, however, this time `t2` will become the last completed
+instant. This design ensures that the regular writers do not fail due to
+indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. Indexer will
+fetch the last instant for which indexing was done from `.index.inflight` file.
Review comment:
would like to understand this more. so the indexer will keep updating the index.inflight meta file is it? like checkpointing ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] Add RFC for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1016589473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4640: [HUDI-3225] Add RFC for async metadata indexing
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r791273720
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
Review comment:
we should support this via regular writers as well. just scheduling. it will make life easier.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
Review comment:
its not very clear what does the indexer does here apart from populating base files for all instants having commit time < t. for eg, what does indexer does to completed instants > t up until now ?
If I am not wrong, "check for all completed commit actions to ensure each of them added entries per its indexing plan" refers to all instants up until now. would be good to call it out.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
Review comment:
in a single write mode, users may not have configured any lock service and we don't enforce one as of today. something to keep in mind.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
Review comment:
also, lets say this is the timeline when indexer started
C1, C2,.... C5 (inflight), C6, C7, C8. Start indexer.
indexer will build base file with all info until C4.
what happens to data pertaining to C6, C7, C8? these are already completed ones.
essentially indexer will go through every completed commit up until now and ensure they are applied to new partition. if its < C= C4, it goes to base file. if its> C4, it goes as delta commit is it ?
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
Review comment:
I don't see details on when exactly regular writers will start to make synchronous updates. Also, when exactly the callers can start using the new index that got built out? whats the source of truth. we can rely on timeline completed instant for the index, but after archival? also loading timeline everytime might be costly as well.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is inflight and
+a basefile is already being written under the metadata partition, then each
+writer will create log files in the same filegroup for the metadata index
+update. This will happen within the existing data table lock. However, if the
+indexing has still not started and instant `t3` is still in requested state,
+then writer will still continue to log entries but indexer will handle this
+scenario and assign the same filegroup id when `t3` transitions to inflight.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out. After
+timeout, indexer will abort writing the instant upto which indexing was done
+in `t3.index.inflight` file in the timeline. At this point, user can trigger the
+index process again, however, this time `t2` will become the last completed
+instant. This design ensures that the regular writers do not fail due to
+indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. Indexer will
+fetch the last instant for which indexing was done from `.index.inflight` file.
+It will start indexing again from the instant thereafter.
+
+**Case 3: Race conditions**
+
+a) Writer went inflight just after an indexing request was added but indexer has
+not yet started executing.
+
+In this case, writer will continue to log updates in metadata partition. At the
+time of execution, indexer will see there are already some log files and handle
+that in the indexing check.
+
+b) Inflight writer about to commit, but indexing completed just before that.
+
+In this case, since the indexer completed before the writer, so it has already
Review comment:
will sync up directly. I don't get this fully.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is inflight and
+a basefile is already being written under the metadata partition, then each
+writer will create log files in the same filegroup for the metadata index
+update. This will happen within the existing data table lock. However, if the
+indexing has still not started and instant `t3` is still in requested state,
+then writer will still continue to log entries but indexer will handle this
+scenario and assign the same filegroup id when `t3` transitions to inflight.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out. After
+timeout, indexer will abort writing the instant upto which indexing was done
+in `t3.index.inflight` file in the timeline. At this point, user can trigger the
+index process again, however, this time `t2` will become the last completed
+instant. This design ensures that the regular writers do not fail due to
+indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. Indexer will
+fetch the last instant for which indexing was done from `.index.inflight` file.
Review comment:
would like to understand this more. so the indexer will keep updating the index.inflight meta file is it? like checkpointing ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1062555481
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347)
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1062554441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347)
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r822837645
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is inflight and
+a basefile is already being written under the metadata partition, then each
+writer will create log files in the same filegroup for the metadata index
+update. This will happen within the existing data table lock. However, if the
+indexing has still not started and instant `t3` is still in requested state,
+then writer will still continue to log entries but indexer will handle this
+scenario and assign the same filegroup id when `t3` transitions to inflight.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out. After
+timeout, indexer will abort writing the instant upto which indexing was done
+in `t3.index.inflight` file in the timeline. At this point, user can trigger the
+index process again, however, this time `t2` will become the last completed
+instant. This design ensures that the regular writers do not fail due to
+indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. Indexer will
+fetch the last instant for which indexing was done from `.index.inflight` file.
Review comment:
No longer valid. To keep things simple, either an MDT partition is available or not available and this will be known through table config. There is some value in checkpointing, especially for indexes that take time, but depending on timeline adds more complexity and we will have to deal with more correctness issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r822830859
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
Review comment:
You mean for inline scheduling like how we have for other table services?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1086042189
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
}, {
"hash" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee05ae22e767b1c62862f9d041ef2835d4667f63",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839)
* ee05ae22e767b1c62862f9d041ef2835d4667f63 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1065151944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r809654236
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
Review comment:
Not sure how the metadata indexing solves the multi-writer problem for MDT. Strictly speaking we just need table service scheduling on MDT by guarded by the lock.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
Review comment:
+1 high level indexer should only write the base files.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
Review comment:
nit: `indexing.requested` ? all actions are verbs
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
Review comment:
We have to clearly document these, along with other operations that cannot be performed without lock provider configured. As safety, should the indexer always error out if there there is no lock provider configured?
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
Review comment:
I think indexer should work upto C8. anything inflight between C4 and C8, has to do the "indexing check". This is a valid case that siva's pointing out.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
Review comment:
Once the indexing action completed, any MDT partition that is currently not being indexed, are considered ready for use
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] Add RFC for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1016640957
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r822833301
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
Review comment:
I think we should error out. I tried to think of a way without taking any lock but we need this minimal locking. We should call it out in documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1065151944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
}, {
"hash" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839",
"triggerID" : "62db921e601f9c81c8bd9bb53df771aec6e2de6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62db921e601f9c81c8bd9bb53df771aec6e2de6e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6839)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r825312103
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,281 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned. The creation of filegroups
+ will be done within a lock.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will not only write
+ base files but also ensure that log entries due to instants after `t` are in
+ the same filegroup i.e. no new filegroup is initialized by writers while
+ indexing is in progress.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes. The indexer will error out if there is no lock provider
+configured.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is pending,
Review comment:
good point, i'll update. basically, we need lock when:
1. creating filegroups while scheduling
2. writing to MDT timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1063194983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] Add RFC for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1016585913
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] Add RFC for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1016589473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r824747770
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,281 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned. The creation of filegroups
+ will be done within a lock.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
Review comment:
not necessarily async. it could be regular writes too. in case of multi-writers, there could be a failed commit waiting to be rolled back.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,281 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned. The creation of filegroups
+ will be done within a lock.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will not only write
+ base files but also ensure that log entries due to instants after `t` are in
+ the same filegroup i.e. no new filegroup is initialized by writers while
+ indexing is in progress.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes. The indexer will error out if there is no lock provider
+configured.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is pending,
Review comment:
In the attached image, I see locks. Would be good to cover whats the critical section for which we acquire lock here for entire design in general.
for eg:
regular writers when checking for pending indexing?
regular writers to check for completed partitions in MDT? (from table config)
async indexer while updating the hoodie table config ?
etc. something like this. I am not claiming we need to acquire lock for all of above. But a list like this would be good to call it out explicitly.
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,281 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned. The creation of filegroups
+ will be done within a lock.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will not only write
+ base files but also ensure that log entries due to instants after `t` are in
+ the same filegroup i.e. no new filegroup is initialized by writers while
+ indexing is in progress.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes. The indexer will error out if there is no lock provider
+configured.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is pending,
+which means indexer has already intialized a filegroup, then each writer will
+create log files in the same filegroup for the metadata index update. This will
+happen within the existing data table lock.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
Review comment:
would like to understand the loop here. I thought we will just go for one round and then timeout. will sync up f2f.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r824640318
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,264 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will schedule INDEX action and add
+ a `<instant_time>.index.requested` to the timeline, which contains the
+ indexing plan. Index scheduling will also initialize the filegroup for
+ the partitions for which indexing is planned.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it. As and when indexing is
+ completed for a partition, then table config (`hoodie.properties`) will
+ be updated to indicate that partition is available for reads or
+ synchronous updates. Hudi table config will be the source of truth for
+ the current state of metadata index.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete (i.e. indexing upto
+ instant `t` is done but before completing indexing commit), it will check for
+ all completed commit instants after `t` to ensure each of them added entries
+ per its indexing plan, otherwise simply abort after a configurable timeout.
+ Let's call this the **indexing check**. So, the indexer will only write base
+ files but ensure that log entries due to instants after `t` are in the same
+ filegroup i.e. no new filegroup is initialized by writers while indexing is
+ in progress.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes. The indexer will error out if there is no lock provider
+configured.
+
+### Multi-writer scenario
+
+
+
+Let us walkthrough a concrete mutli-writer scenario to understand the above
+indexing mechanism. In this scenario, let instant `t0` be the last completed
+instant on the timeline. Suppose user triggered index building from an external
+process at `t3`. This will create `t3.index.requested` file with the indexing
+plan. The plan contains the metadata partitions that need to be created and the
+last completed instant, e.g.
+
+```
+[
+ {MetadataPartitionType.FILES.partitionPath(), t0},
+ {MetadataPartitionType.BLOOM_FILTER.partitionPath(), t0},
+ {MetadataPartitionType.COLUMN_STATS.partitionPath(), t0}
+]
+```
+
+Further, suppose there were two inflight writers Writer1 and Writer2 (with
+inflight instants `t1` and `t2` respectively) while the indexing was requested
+or inflight. In this case, the writers will check for pending index action and
+find a pending instant `t3`. Now, if the metadata index creation is pending,
+which means indexer has already intialized a filegroup, then each writer will
+create log files in the same filegroup for the metadata index update. This will
+happen within the existing data table lock.
+
+The indexer runs in a loop until the metadata for data upto `t0` plus the data
+written due to `t1` and `t2` has been indexed, or the indexing timed out.
+Whether indexing timed out or not, table config would be updated with any MDT
+partition(s) for which indexing was complete till `t2`. In case of timeout
+indexer will abort. At this point, user can trigger the index process again,
+however, this time indexer will check for available partitions in table config
+and skip those partitions. This design ensures that the regular writers do not
+fail due to indexing.
+
+### Error Handling
+
+**Case 1: Writer fails while indexer is inflight**
+
+This means index update due to writer did not complete. Indexer continues to
+build the index ignoring the failed instant due to writer. The next update by
+the writer will trigger a rollback of the failed instant, which will also
+rollback incomplete updates in metadata table.
+
+**Case 2: Indexer fails while writer is inflight**
+
+Writer will commit adding log entries to the metadata partition. However, table
+config will indicate that partition is not ready to use. When indexer is
+re-triggered, it will check the plan and table config to figure out which MDT
+partitions to index and start indexing for those partitions.
+
+**Case 3: Race conditions**
Review comment:
Good point! Intialization fo filegroups happen when index is scheduled. While scheduling we can take a lock. I'll update in RFC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r822833726
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
+ 1. The corner case here would be that the indexing check does not factor in
+ the inflight writer just about to commit. But given indexing would take
+ some finite amount of time to go from requested to completion (or we can
+ add some, configurable artificial delays here say 60 seconds), an
+ inflight writer, that is just about to commit concurrently, has a very
+ high chance of seeing the indexing plan and aborting itself.
+
+We can just introduce a lock for adding events to the timeline and these races
+would vanish completely, still providing great scalability and asynchrony for
+these processes.
+
Review comment:
Added some more details. Table config will be the source of truth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1016640957
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r822835191
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
+ up to instant time `t`, where `t` is the latest completed instant time on
+ the timeline without any
+ "holes" i.e. no pending async operations prior to it.
+ 3. The indexing process will write these out as base files within the
+ corresponding metadata partition. A metadata partition cannot be used if
+ there is any pending indexing action against it.
+
+2. Any inflight writers (i.e. with instant time `t'` > `t`) will check for any
+ new indexing request on the timeline prior to preparing to commit.
+ 1. Such writers will proceed to additionally add log entries corresponding
+ to each such indexing request into the metadata partition.
+ 2. There is always a TOCTOU issue here, where the inflight writer may not
+ see an indexing request that was just added and proceed to commit without
+ that. We will correct this during indexing action completion. In the
+ average case, this may not happen and the design has liveness.
+
+3. When the indexing process is about to complete, it will check for all
+ completed commit actions to ensure each of them added entries per its
+ indexing plan, otherwise simply abort after a configurable timeout. Let's
+ call this the **indexing check**.
Review comment:
That's correct and if the indexer times out before C5 completes then it will abort. Next time, indexing will start again with C4 as base instant and run indexing check again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] Add RFC for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1016585913
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#discussion_r824647337
##########
File path: rfc/rfc-45/rfc-45.md
##########
@@ -0,0 +1,229 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# RFC-45: Asynchronous Metadata Indexing
+
+## Proposers
+
+- @codope
+- @manojpec
+
+## Approvers
+
+- @nsivabalan
+- @vinothchandar
+
+## Status
+
+JIRA: [HUDI-2488](https://issues.apache.org/jira/browse/HUDI-2488)
+
+## Abstract
+
+Metadata indexing (aka metadata bootstrapping) is the process of creation of one
+or more metadata-based indexes, e.g. data partitions to files index, that is
+stored in Hudi metadata table. Currently, the metadata table (referred as MDT
+hereafter) supports single partition which is created synchronously with the
+corresponding data table, i.e. commits are first applied to metadata table
+followed by data table. Our goal for MDT is to support multiple partitions to
+boost the performance of existing index and records lookup. However, the
+synchronous manner of metadata indexing is not very scalable as we add more
+partitions to the MDT because the regular writers (writing to the data table)
+have to wait until the MDT commit completes. In this RFC, we propose a design to
+support asynchronous metadata indexing.
+
+## Background
+
+We can read more about the MDT design
+in [RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements)
+. Here is a quick summary of the current state (Hudi v0.10.1). MDT is an
+internal Merge-on-Read (MOR) table that has a single partition called `files`
+which stores the data partitions to files index that is used in file listing.
+MDT is co-located with the data table (inside `.hoodie/metadata` directory under
+the basepath). In order to handle multi-writer scenario, users configure lock
+provider and only one writer can access MDT in read-write mode. Hence, any write
+to MDT is guarded by the data table lock. This ensures only one write is
+committed to MDT at any point in time and thus guarantees serializability.
+However, locking overhead adversely affects the write throughput and will reach
+its scalability limits as we add more partitions to the MDT.
+
+## Goals
+
+- Support indexing one or more partitions in MDT while regular writers and table
+ services (such as cleaning or compaction) are in progress.
+- Locking to be as lightweight as possible.
+- Keep required config changes to a minimum to simplify deployment / upgrade in
+ production.
+- Do not require specific ordering of how writers and table service pipelines
+ need to be upgraded / restarted.
+- If an external long-running process is being used to initialize the index, the
+ process should be made idempotent so it can handle errors from previous runs.
+- To re-initialize the index, make it as simple as running the external
+ initialization process again without having to change configs.
+
+## Implementation
+
+### A new Hudi action: INDEX
+
+We introduce a new action `index` which will denote the index building process,
+the mechanics of which is as follows:
+
+1. From an external process, users can issue a CREATE INDEX or similar statement
+ to trigger indexing for an existing table.
+ 1. This will add a `<instant_time>.index.requested` to the timeline, which
+ contains the indexing plan.
+ 2. From here on, the index building process will continue to build an index
Review comment:
Yes we can do that and can avoid little serde cost. It can also ease debugging. However, i should point out that index action will be written on the data timeline as it will be known to the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1063102102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
}, {
"hash" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750",
"triggerID" : "7afccec9740814a4bf7f8a3d8a6125c223829d27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719)
* 7afccec9740814a4bf7f8a3d8a6125c223829d27 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6750)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1062584118
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1063097604
@vinothchandar @nsivabalan This is ready for review again. Following has changed since the last review:
- Filegroup in metadata partition will be initialized while scheduling index action. No new filegroup is initialized by writers while indexing is in progress.
- Table config will be the source of truth for what metadata partitions are available, instead of relying on the commit metadata on timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4640: [HUDI-3225] [RFC-45] for async metadata indexing
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4640:
URL: https://github.com/apache/hudi/pull/4640#issuecomment-1062555481
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347",
"triggerID" : "1cb4c545b9d25ab018770887fa619654a486b0c2",
"triggerType" : "PUSH"
}, {
"hash" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719",
"triggerID" : "bd3b3541599ce16f91b73d4923ad7f875f4b3d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1cb4c545b9d25ab018770887fa619654a486b0c2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5347)
* bd3b3541599ce16f91b73d4923ad7f875f4b3d2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6719)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org