You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2024/12/04 23:09:48 UTC
[PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
yihua opened a new pull request, #12424:
URL: https://github.com/apache/hudi/pull/12424
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2518746833
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e056a36a6cfc2934315ea21a0be44567acd1d6b9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12424:
URL: https://github.com/apache/hudi/pull/12424#discussion_r1870549190
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##########
@@ -161,4 +162,20 @@ protected final void dropIndexOnRestore() {
}
}
}
+
+ protected final void maybeDisableWriteRecordPositions() {
Review Comment:
can you just do it in the writeClient.upsert ? The entrance of the DAG trigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2519031354
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
}, {
"hash" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281",
"triggerID" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"triggerType" : "PUSH"
}, {
"hash" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df31c71abc97348352d2fb31a47132f2aef6c221 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281)
* 8c848a0c1c735e79546e19cd22133da519445b7b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2518954859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f13883a7721beb869cca070f7c3b5c910ea62002 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272)
* 8d281666ebf4cea35a3646dcc39cdd0c3280e555 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation with NBCC or during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2519257341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
}, {
"hash" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281",
"triggerID" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"triggerType" : "PUSH"
}, {
"hash" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"triggerType" : "PUSH"
}, {
"hash" : "d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2284",
"triggerID" : "d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c848a0c1c735e79546e19cd22133da519445b7b UNKNOWN
* d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2284)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation with NBCC or during pending compaction [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2519308203
<img width="1216" alt="image" src="https://github.com/user-attachments/assets/b3f1f542-314d-4c7e-b169-e2dae92ae5c7">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12424:
URL: https://github.com/apache/hudi/pull/12424#discussion_r1870507307
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##########
@@ -161,4 +161,15 @@ protected final void dropIndexOnRestore() {
}
}
}
+
+ protected final void disableWriteRecordPositionsWithPendingCompaction() {
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2518949115
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f13883a7721beb869cca070f7c3b5c910ea62002 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272)
* 8d281666ebf4cea35a3646dcc39cdd0c3280e555 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12424:
URL: https://github.com/apache/hudi/pull/12424#discussion_r1870545651
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##########
@@ -161,4 +162,20 @@ protected final void dropIndexOnRestore() {
}
}
}
+
+ protected final void maybeDisableWriteRecordPositions() {
Review Comment:
Doing it at the `HoodieAppendHandle` level requires instantiating and scanning the timeline to find the compaction instant at every executor, which is an overhead that should not be introduced I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation with NBCC or during pending compaction [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #12424:
URL: https://github.com/apache/hudi/pull/12424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12424:
URL: https://github.com/apache/hudi/pull/12424#discussion_r1870576461
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##########
@@ -161,4 +162,20 @@ protected final void dropIndexOnRestore() {
}
}
}
+
+ protected final void maybeDisableWriteRecordPositions() {
Review Comment:
Moved the write config modification to the `BaseHoodieWriteClient` and only exercise it in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2519030108
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
}, {
"hash" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281",
"triggerID" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"triggerType" : "PUSH"
}, {
"hash" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8d281666ebf4cea35a3646dcc39cdd0c3280e555 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275)
* df31c71abc97348352d2fb31a47132f2aef6c221 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281)
* 8c848a0c1c735e79546e19cd22133da519445b7b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2519028780
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
}, {
"hash" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f13883a7721beb869cca070f7c3b5c910ea62002 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272)
* 8d281666ebf4cea35a3646dcc39cdd0c3280e555 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275)
* df31c71abc97348352d2fb31a47132f2aef6c221 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation with NBCC or during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2519099467
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
}, {
"hash" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281",
"triggerID" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"triggerType" : "PUSH"
}, {
"hash" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"triggerType" : "PUSH"
}, {
"hash" : "d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df31c71abc97348352d2fb31a47132f2aef6c221 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281)
* 8c848a0c1c735e79546e19cd22133da519445b7b UNKNOWN
* d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2518921149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e056a36a6cfc2934315ea21a0be44567acd1d6b9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269)
* f13883a7721beb869cca070f7c3b5c910ea62002 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12424:
URL: https://github.com/apache/hudi/pull/12424#discussion_r1870538411
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##########
@@ -161,4 +162,20 @@ protected final void dropIndexOnRestore() {
}
}
}
+
+ protected final void maybeDisableWriteRecordPositions() {
Review Comment:
I don't think we should fix it in the executors, because COW table also share these executors. I kind of think we should fix it in the `HoodieAppendHandle`.
```java
this.shouldWriteRecordPositions = config.shouldWriteRecordPositions()
// record positions supported only from table version 8
&& config.getWriteVersion().greaterThanOrEquals(HoodieTableVersion.EIGHT);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12424:
URL: https://github.com/apache/hudi/pull/12424#discussion_r1870548962
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##########
@@ -161,4 +162,20 @@ protected final void dropIndexOnRestore() {
}
}
}
+
+ protected final void maybeDisableWriteRecordPositions() {
Review Comment:
can you just do it in the writeClient.upsert ? The entrance of the DAG trigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12424:
URL: https://github.com/apache/hudi/pull/12424#discussion_r1870545651
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##########
@@ -161,4 +162,20 @@ protected final void dropIndexOnRestore() {
}
}
}
+
+ protected final void maybeDisableWriteRecordPositions() {
Review Comment:
Doing it at the `HoodieAppendHandle` level requires instantiating and scanning the timeline at every executor, which is an overhead that should not be introduced I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2518919564
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e056a36a6cfc2934315ea21a0be44567acd1d6b9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269)
* f13883a7721beb869cca070f7c3b5c910ea62002 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2518744785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e056a36a6cfc2934315ea21a0be44567acd1d6b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12424:
URL: https://github.com/apache/hudi/pull/12424#discussion_r1870456171
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/BaseActionExecutor.java:
##########
@@ -161,4 +161,15 @@ protected final void dropIndexOnRestore() {
}
}
}
+
+ protected final void disableWriteRecordPositionsWithPendingCompaction() {
Review Comment:
we also need to disable write positions when nbcc is enabled.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSourceStorage.scala:
##########
@@ -184,4 +190,125 @@ class TestMORDataSourceStorage extends SparkClientFunctionalTestHarness {
val metaClient = HoodieTestUtils.createMetaClient(new HadoopStorageConfiguration(fs.getConf), basePath)
assertEquals(1, metaClient.getActiveTimeline.getCommitAndReplaceTimeline.countInstants())
}
+
+ @Test
+ def testAutoDisablingRecordPositionsUnderPendingCompaction(): Unit = {
+ val options = Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ "hoodie.bulkinsert.shuffle.parallelism" -> "2",
+ "hoodie.delete.shuffle.parallelism" -> "1",
+ "hoodie.merge.small.file.group.candidates.limit" -> "0",
+ HoodieWriteConfig.WRITE_RECORD_POSITIONS.key -> "true",
+ DataSourceWriteOptions.RECORDKEY_FIELD.key -> "_row_key",
+ DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key -> classOf[NonpartitionedKeyGenerator].getName,
+ DataSourceWriteOptions.PRECOMBINE_FIELD.key -> "timestamp",
+ HoodieWriteConfig.TBL_NAME.key -> "hoodie_test"
+ )
+ val optionWithoutCompactionExecution = options ++ Map(
Review Comment:
nice. I did not think we can leverage this in tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2518843495
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e056a36a6cfc2934315ea21a0be44567acd1d6b9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation with NBCC or during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2519103166
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2275",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
}, {
"hash" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281",
"triggerID" : "df31c71abc97348352d2fb31a47132f2aef6c221",
"triggerType" : "PUSH"
}, {
"hash" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c848a0c1c735e79546e19cd22133da519445b7b",
"triggerType" : "PUSH"
}, {
"hash" : "d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2284",
"triggerID" : "d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df31c71abc97348352d2fb31a47132f2aef6c221 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2281)
* 8c848a0c1c735e79546e19cd22133da519445b7b UNKNOWN
* d31ee26ebfdb265e1a4c3847a9e6dc09ce0aef17 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2284)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8653] Disable record position generation during pending compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12424:
URL: https://github.com/apache/hudi/pull/12424#issuecomment-2518947507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269",
"triggerID" : "e056a36a6cfc2934315ea21a0be44567acd1d6b9",
"triggerType" : "PUSH"
}, {
"hash" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272",
"triggerID" : "f13883a7721beb869cca070f7c3b5c910ea62002",
"triggerType" : "PUSH"
}, {
"hash" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d281666ebf4cea35a3646dcc39cdd0c3280e555",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e056a36a6cfc2934315ea21a0be44567acd1d6b9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2269)
* f13883a7721beb869cca070f7c3b5c910ea62002 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2272)
* 8d281666ebf4cea35a3646dcc39cdd0c3280e555 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org