You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "amrishlal (via GitHub)" <gi...@apache.org> on 2023/07/15 06:20:55 UTC
[GitHub] [hudi] amrishlal opened a new pull request, #9203: [HUDI-6315] [WIP] Feature flag for disabling prepped merge.
amrishlal opened a new pull request, #9203:
URL: https://github.com/apache/hudi/pull/9203
### Change Logs
Add user-defined feature flag for disabling prepped merge.
### Impact
New feature flag `ENABLE_OPTIMIZED_MERGE_WRITES`
### Risk level (write none, low medium or high below)
Low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [X] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [X] Change Logs and Impact were stated clearly
- [X] Adequate tests were added if applicable
- [X] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1640483825
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268783724
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -313,7 +313,7 @@ object DataSourceWriteOptions {
/**
* Config key with boolean value that indicates whether record being written is already prepped.
*/
- val DATASOURCE_WRITE_PREPPED_KEY = "_hoodie.datasource.write.prepped";
+ val SPARK_SQL_WRITE_PREPPED_KEY = "_hoodie.spark.sql.writes.prepped";
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1267607758
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -721,6 +721,8 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final String WRITE_PREPPED_MERGE_KEY = "_hoodie.datasource.merge.into.prepped";
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1641450952
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669)
* 539cad2f3b8edde2211bd8ddeeb3feec15cd6e94 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1641848393
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbecc75443a72ac360f87b80913837c32d62c466 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1642359770
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #9203:
URL: https://github.com/apache/hudi/pull/9203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] [WIP] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1265936387
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieInternalConfig.java:
##########
@@ -46,6 +46,13 @@ public class HoodieInternalConfig extends HoodieConfig {
.withDocumentation("For SQL operations, if enables bulk_insert operation, "
+ "this configure will take effect to decide overwrite whole table or partitions specified");
+ public static final ConfigProperty<Boolean> ENABLE_PREPPED_MERGE_WRITES = ConfigProperty
+ .key("hoodie.internal.sql.prepped.merge.enabled")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("0.14.0")
+ .withDocumentation("Controls whether spark sql optimized merge is enabled.");
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #9203: [HUDI-6315] [WIP] Feature flag for disabling prepped merge.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1640349773
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268914202
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -721,6 +721,8 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final String SPARK_SQL_MERGE_INTO_PREPPED_KEY = "_hoodie.spark.sql.merge.into.prepped";
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1643176885
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
}, {
"hash" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704",
"triggerID" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"triggerType" : "PUSH"
}, {
"hash" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18712",
"triggerID" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cad8c2a2950628a8da9d0834735fbdb29871c85d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704)
* 0af9136d05a06e9a654f4c738d41a9063bdc1b00 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18712)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268827158
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -82,7 +82,7 @@ object HoodieWriterUtils {
hoodieConfig.setDefaultValue(RECONCILE_SCHEMA)
hoodieConfig.setDefaultValue(DROP_PARTITION_COLUMNS)
hoodieConfig.setDefaultValue(KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED)
- hoodieConfig.setDefaultValue(ENABLE_OPTIMIZED_SQL_WRITES)
+ hoodieConfig.setDefaultValue(SPARK_SQL_OPTIMIZED_WRITES)
Review Comment:
Removed. From what I can see it is not really needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1641576210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 539cad2f3b8edde2211bd8ddeeb3feec15cd6e94 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679)
* cbecc75443a72ac360f87b80913837c32d62c466 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1640477276
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1643508441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
}, {
"hash" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704",
"triggerID" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"triggerType" : "PUSH"
}, {
"hash" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18712",
"triggerID" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0af9136d05a06e9a654f4c738d41a9063bdc1b00 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18712)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268662043
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/SparkHoodieIndexFactory.java:
##########
@@ -44,8 +43,7 @@
*/
public final class SparkHoodieIndexFactory {
public static HoodieIndex createIndex(HoodieWriteConfig config) {
- boolean mergeIntoWrites = config.getProps().getBoolean(HoodieInternalConfig.SQL_MERGE_INTO_WRITES.key(),
- HoodieInternalConfig.SQL_MERGE_INTO_WRITES.defaultValue());
+ boolean mergeIntoWrites = config.getProps().getBoolean(HoodieWriteConfig.SPARK_SQL_MERGE_INTO_PREPPED_KEY, false);
Review Comment:
lets align the var name w/ the config.
boolean sqlMergeIntoPrepped
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieWriterUtils.scala:
##########
@@ -82,7 +82,7 @@ object HoodieWriterUtils {
hoodieConfig.setDefaultValue(RECONCILE_SCHEMA)
hoodieConfig.setDefaultValue(DROP_PARTITION_COLUMNS)
hoodieConfig.setDefaultValue(KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED)
- hoodieConfig.setDefaultValue(ENABLE_OPTIMIZED_SQL_WRITES)
+ hoodieConfig.setDefaultValue(SPARK_SQL_OPTIMIZED_WRITES)
Review Comment:
lets check if this is really required.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -348,9 +356,8 @@ object HoodieSparkSqlWriter {
}
// Remove meta columns from writerSchema if isPrepped is true.
- val isPrepped = hoodieConfig.getBooleanOrDefault(DATASOURCE_WRITE_PREPPED_KEY, false)
- val mergeIntoWrites = parameters.getOrDefault(SQL_MERGE_INTO_WRITES.key(),
- SQL_MERGE_INTO_WRITES.defaultValue.toString).toBoolean
+ val isPrepped = hoodieConfig.getBooleanOrDefault(SPARK_SQL_WRITE_PREPPED_KEY, false)
+ val mergeIntoWrites = parameters.getOrDefault(SPARK_SQL_MERGE_INTO_PREPPED_KEY, "false").toBoolean
Review Comment:
lets fix the var name
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -313,7 +313,7 @@ object DataSourceWriteOptions {
/**
* Config key with boolean value that indicates whether record being written is already prepped.
*/
- val DATASOURCE_WRITE_PREPPED_KEY = "_hoodie.datasource.write.prepped";
+ val SPARK_SQL_WRITE_PREPPED_KEY = "_hoodie.spark.sql.writes.prepped";
Review Comment:
minor
"SPARK_SQL_WRITES_PREPPED_KEY"
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestDeleteTable.scala:
##########
@@ -47,7 +47,7 @@ class TestDeleteTable extends HoodieSparkSqlTestBase {
""".stripMargin)
// test with optimized sql writes enabled / disabled.
- spark.sql(s"set hoodie.spark.sql.writes.optimized.enable=$optimizedSqlEnabled")
+ spark.sql(s"set hoodie.spark.sql.optimized.writes.enable=$optimizedSqlEnabled")
Review Comment:
lets see if we can use the variable to avoid any mis-steps
##########
hudi-spark-datasource/hudi-spark/src/test/java/HoodieJavaApp.java:
##########
@@ -161,7 +161,7 @@ public void run() throws Exception {
.option(DataSourceWriteOptions.ASYNC_COMPACT_ENABLE().key(), "false")
.option(DataSourceWriteOptions.ASYNC_CLUSTERING_ENABLE().key(), "true")
.option(DataSourceWriteOptions.ASYNC_CLUSTERING_ENABLE().key(), "true")
- .option(DataSourceWriteOptions.ENABLE_OPTIMIZED_SQL_WRITES().key(), "true")
+ .option(DataSourceWriteOptions.SPARK_SQL_OPTIMIZED_WRITES().key(), "true")
Review Comment:
for defaults, lets try to use the actual default instead of hard coding it
DataSourceWriteOptions.SPARK_SQL_OPTIMIZED_WRITES().default()
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -617,6 +622,8 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
val hiveSyncConfig = buildHiveSyncConfig(sparkSession, hoodieCatalogTable, tableConfig)
+ val enableOptimizedMerge = sparkSession.sqlContext.conf.getConfString(SPARK_SQL_OPTIMIZED_WRITES.key(),
+ SPARK_SQL_OPTIMIZED_WRITES.defaultValue())
Review Comment:
lets introduce
val keyGeneratorClassName = ....
and use it in L 636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1642980886
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
}, {
"hash" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbecc75443a72ac360f87b80913837c32d62c466 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691)
* cad8c2a2950628a8da9d0834735fbdb29871c85d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1640555900
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1640372277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on pull request #9203: [HUDI-6315] [WIP] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1639487695
>Lets also parametrize some tests for MIT as well
Parameterized test case to test with both `hoodie.spark.sql.optimized.merge.enable` set to `true` and `false`
> We can set ENABLE_PREPPED_MERGE_WRITES in MergeIntoHoodieTableCommand using hoodie.spark.sql.writes.optimized.enable. We don't need to introduce another public config that way
Moved `SQL_MERGE_INTO_WRITES` back to `HoodieSparkSqlWriter.scala` as it was originally, created `ENABLE_OPTIMIZED_SQL_MERGE_WRITES` in `DataSourceOptions.scala` for use as public config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268789337
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestDeleteTable.scala:
##########
@@ -47,7 +47,7 @@ class TestDeleteTable extends HoodieSparkSqlTestBase {
""".stripMargin)
// test with optimized sql writes enabled / disabled.
- spark.sql(s"set hoodie.spark.sql.writes.optimized.enable=$optimizedSqlEnabled")
+ spark.sql(s"set hoodie.spark.sql.optimized.writes.enable=$optimizedSqlEnabled")
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1644022239
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1644776087
CI failed due to known flaky tests. going ahead w/ merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1267608249
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTableWithNonRecordKeyField.scala:
##########
@@ -22,122 +22,128 @@ import org.apache.hudi.{HoodieSparkUtils, ScalaAssertionSupport}
class TestMergeIntoTableWithNonRecordKeyField extends HoodieSparkSqlTestBase with ScalaAssertionSupport {
test("Test Merge into extra cond") {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts long
- |) using hudi
- | location '${tmp.getCanonicalPath}/$tableName'
- | tblproperties (
- | primaryKey ='id',
- | preCombineField = 'ts'
- | )
+ Seq(true, false).foreach { optimizedSqlEnabled =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey ='id',
+ | preCombineField = 'ts'
+ | )
""".stripMargin)
- val tableName2 = generateTableName
- spark.sql(
- s"""
- |create table $tableName2 (
- | id int,
- | name string,
- | price double,
- | ts long
- |) using hudi
- | location '${tmp.getCanonicalPath}/$tableName2'
- | tblproperties (
- | primaryKey ='id',
- | preCombineField = 'ts'
- | )
+ val tableName2 = generateTableName
+ spark.sql(
+ s"""
+ |create table $tableName2 (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | location '${tmp.getCanonicalPath}/$tableName2'
+ | tblproperties (
+ | primaryKey ='id',
+ | preCombineField = 'ts'
+ | )
""".stripMargin)
- spark.sql(
- s"""
- |insert into $tableName values
- | (1, 'a1', 10, 100),
- | (2, 'a2', 20, 200),
- | (3, 'a3', 20, 100)
- |""".stripMargin)
- spark.sql(
- s"""
- |insert into $tableName2 values
- | (1, 'u1', 10, 999),
- | (3, 'u3', 30, 9999),
- | (4, 'u4', 40, 99999)
- |""".stripMargin)
+ spark.sql(
+ s"""
+ |insert into $tableName values
+ | (1, 'a1', 10, 100),
+ | (2, 'a2', 20, 200),
+ | (3, 'a3', 20, 100)
+ |""".stripMargin)
+ spark.sql(
+ s"""
+ |insert into $tableName2 values
+ | (1, 'u1', 10, 999),
+ | (3, 'u3', 30, 9999),
+ | (4, 'u4', 40, 99999)
+ |""".stripMargin)
- spark.sql(
- s"""
- |merge into $tableName as oldData
- |using $tableName2
- |on oldData.id = $tableName2.id
- |when matched and oldData.price = $tableName2.price then update set oldData.name = $tableName2.name
- |
- |""".stripMargin)
+ // test with optimized sql merge enabled / disabled.
+ spark.sql(s"set hoodie.spark.sql.optimized.merge.enable=$optimizedSqlEnabled")
- checkAnswer(s"select id, name, price, ts from $tableName")(
- Seq(1, "u1", 10.0, 100),
- Seq(3, "a3", 20.0, 100),
- Seq(2, "a2", 20.0, 200)
- )
+ spark.sql(
+ s"""
+ |merge into $tableName as oldData
+ |using $tableName2
+ |on oldData.id = $tableName2.id
+ |when matched and oldData.price = $tableName2.price then update set oldData.name = $tableName2.name
+ |
+ |""".stripMargin)
- val errorMessage = if (HoodieSparkUtils.gteqSpark3_1) {
- "Only simple conditions of the form `t.id = s.id` using primary key or partition path " +
- "columns are allowed on tables with primary key. (illegal column(s) used: `price`"
- } else {
- "Only simple conditions of the form `t.id = s.id` using primary key or partition path " +
- "columns are allowed on tables with primary key. (illegal column(s) used: `price`;"
- }
+ checkAnswer(s"select id, name, price, ts from $tableName")(
+ Seq(1, "u1", 10.0, 100),
+ Seq(3, "a3", 20.0, 100),
+ Seq(2, "a2", 20.0, 200)
+ )
- checkException(
- s"""
- |merge into $tableName as oldData
- |using $tableName2
- |on oldData.id = $tableName2.id and oldData.price = $tableName2.price
- |when matched then update set oldData.name = $tableName2.name
- |when not matched then insert *
- |""".stripMargin)(errorMessage)
+ val errorMessage = if (HoodieSparkUtils.gteqSpark3_1) {
+ "Only simple conditions of the form `t.id = s.id` using primary key or partition path " +
+ "columns are allowed on tables with primary key. (illegal column(s) used: `price`"
+ } else {
+ "Only simple conditions of the form `t.id = s.id` using primary key or partition path " +
+ "columns are allowed on tables with primary key. (illegal column(s) used: `price`;"
+ }
- //test with multiple pks
- val tableName3 = generateTableName
- spark.sql(
- s"""
- |create table $tableName3 (
- | id int,
- | name string,
- | price double,
- | ts long
- |) using hudi
- | location '${tmp.getCanonicalPath}/$tableName3'
- | tblproperties (
- | primaryKey ='id,name',
- | preCombineField = 'ts'
- | )
+ checkException(
+ s"""
+ |merge into $tableName as oldData
+ |using $tableName2
+ |on oldData.id = $tableName2.id and oldData.price = $tableName2.price
+ |when matched then update set oldData.name = $tableName2.name
+ |when not matched then insert *
+ |""".stripMargin)(errorMessage)
+
+ //test with multiple pks
+ val tableName3 = generateTableName
+ spark.sql(
+ s"""
+ |create table $tableName3 (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | location '${tmp.getCanonicalPath}/$tableName3'
+ | tblproperties (
+ | primaryKey ='id,name',
+ | preCombineField = 'ts'
+ | )
""".stripMargin)
- val errorMessage2 = if (HoodieSparkUtils.gteqSpark3_1) {
- "Hudi tables with primary key are required to match on all primary key colums. Column: 'name' not found"
- } else {
- "Hudi tables with primary key are required to match on all primary key colums. Column: 'name' not found;"
- }
+ val errorMessage2 = if (HoodieSparkUtils.gteqSpark3_1) {
+ "Hudi tables with primary key are required to match on all primary key colums. Column: 'name' not found"
+ } else {
+ "Hudi tables with primary key are required to match on all primary key colums. Column: 'name' not found;"
+ }
- checkException(
- s"""
- |merge into $tableName3 as oldData
- |using $tableName2
- |on oldData.id = $tableName2.id
- |when matched then update set oldData.name = $tableName2.name
- |when not matched then insert *
- |""".stripMargin)(errorMessage2)
+ checkException(
+ s"""
+ |merge into $tableName3 as oldData
+ |using $tableName2
+ |on oldData.id = $tableName2.id
+ |when matched then update set oldData.name = $tableName2.name
+ |when not matched then insert *
+ |""".stripMargin)(errorMessage2)
+ }
}
}
test("Test pkless complex merge cond") {
withRecordType()(withTempDir { tmp =>
spark.sql("set hoodie.payload.combined.schema.validate = true")
+ spark.sql("set hoodie.spark.sql.optimized.merge.enable=true")
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1642403069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* cbecc75443a72ac360f87b80913837c32d62c466 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1640897010
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268819905
##########
hudi-spark-datasource/hudi-spark/src/test/java/HoodieJavaApp.java:
##########
@@ -161,7 +161,7 @@ public void run() throws Exception {
.option(DataSourceWriteOptions.ASYNC_COMPACT_ENABLE().key(), "false")
.option(DataSourceWriteOptions.ASYNC_CLUSTERING_ENABLE().key(), "true")
.option(DataSourceWriteOptions.ASYNC_CLUSTERING_ENABLE().key(), "true")
- .option(DataSourceWriteOptions.ENABLE_OPTIMIZED_SQL_WRITES().key(), "true")
+ .option(DataSourceWriteOptions.SPARK_SQL_OPTIMIZED_WRITES().key(), "true")
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1640493692
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268783937
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -348,9 +356,8 @@ object HoodieSparkSqlWriter {
}
// Remove meta columns from writerSchema if isPrepped is true.
- val isPrepped = hoodieConfig.getBooleanOrDefault(DATASOURCE_WRITE_PREPPED_KEY, false)
- val mergeIntoWrites = parameters.getOrDefault(SQL_MERGE_INTO_WRITES.key(),
- SQL_MERGE_INTO_WRITES.defaultValue.toString).toBoolean
+ val isPrepped = hoodieConfig.getBooleanOrDefault(SPARK_SQL_WRITE_PREPPED_KEY, false)
+ val mergeIntoWrites = parameters.getOrDefault(SPARK_SQL_MERGE_INTO_PREPPED_KEY, "false").toBoolean
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268877033
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -721,6 +721,8 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final String SPARK_SQL_MERGE_INTO_PREPPED_KEY = "_hoodie.spark.sql.merge.into.prepped";
Review Comment:
lets add java docs
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -648,7 +662,7 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
RECONCILE_SCHEMA.key -> "false",
CANONICALIZE_NULLABLE.key -> "false",
SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.key -> "true",
- HoodieInternalConfig.SQL_MERGE_INTO_WRITES.key -> "true",
+ HoodieWriteConfig.SPARK_SQL_MERGE_INTO_PREPPED_KEY -> enableOptimizedMerge,
Review Comment:
we should see some test failures for sure.
can you check for tests added in this patch https://github.com/apache/hudi/pull/6358
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -648,7 +662,7 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
RECONCILE_SCHEMA.key -> "false",
CANONICALIZE_NULLABLE.key -> "false",
SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.key -> "true",
- HoodieInternalConfig.SQL_MERGE_INTO_WRITES.key -> "true",
+ HoodieWriteConfig.SPARK_SQL_MERGE_INTO_PREPPED_KEY -> enableOptimizedMerge,
Review Comment:
I guess we have to set SQL_MERGE_INTO_WRITES.key -> "true" as well here in addition to setting HoodieWriteConfig.SPARK_SQL_MERGE_INTO_PREPPED_KEY
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1641563823
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669)
* 539cad2f3b8edde2211bd8ddeeb3feec15cd6e94 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679)
* cbecc75443a72ac360f87b80913837c32d62c466 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on pull request #9203: [HUDI-6315] [WIP] Feature flag for disabling prepped merge.
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1638075596
We can set ENABLE_PREPPED_MERGE_WRITES in MergeIntoHoodieTableCommand using hoodie.spark.sql.writes.optimized.enable. We don't need to introduce another public config that way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1267481558
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -721,6 +721,8 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final String WRITE_PREPPED_MERGE_KEY = "_hoodie.datasource.merge.prepped";
+
Review Comment:
I am also thinking, from a user standpoint we should have just 1 config to enable or disable the optimized flow (irrespective of whether its mIT or updates or deletes).
but internally we can use diff configs if we wish to differentiate MIT and rest.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -642,11 +642,18 @@ object DataSourceWriteOptions {
val DROP_PARTITION_COLUMNS: ConfigProperty[java.lang.Boolean] = HoodieTableConfig.DROP_PARTITION_COLUMNS
val ENABLE_OPTIMIZED_SQL_WRITES: ConfigProperty[String] = ConfigProperty
- .key("hoodie.spark.sql.writes.optimized.enable")
+ .key("hoodie.spark.sql.optimized.writes.enable")
.defaultValue("true")
.markAdvanced()
.sinceVersion("0.14.0")
- .withDocumentation("Controls whether spark sql optimized update is enabled.")
+ .withDocumentation("Controls whether spark sql prepped update and delete is enabled.")
+
+ val ENABLE_OPTIMIZED_SQL_MERGE_WRITES: ConfigProperty[String] = ConfigProperty
Review Comment:
I am also thinking, from a user standpoint we should have just 1 config to enable or disable the optimized flow (irrespective of whether its mIT or updates or deletes).
but internally we can use diff configs if we wish to differentiate MIT and rest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1640457527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9203: [HUDI-6315] [WIP] Feature flag for disabling prepped merge.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1264768190
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieInternalConfig.java:
##########
@@ -46,6 +46,13 @@ public class HoodieInternalConfig extends HoodieConfig {
.withDocumentation("For SQL operations, if enables bulk_insert operation, "
+ "this configure will take effect to decide overwrite whole table or partitions specified");
+ public static final ConfigProperty<Boolean> ENABLE_PREPPED_MERGE_WRITES = ConfigProperty
+ .key("hoodie.internal.sql.prepped.merge.enabled")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("0.14.0")
+ .withDocumentation("Controls whether spark sql optimized merge is enabled.");
Review Comment:
lets name the the config
hoodie.spark.sql.optimized.merge.enable
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieInternalConfig.java:
##########
@@ -46,6 +46,13 @@ public class HoodieInternalConfig extends HoodieConfig {
.withDocumentation("For SQL operations, if enables bulk_insert operation, "
+ "this configure will take effect to decide overwrite whole table or partitions specified");
+ public static final ConfigProperty<Boolean> ENABLE_PREPPED_MERGE_WRITES = ConfigProperty
+ .key("hoodie.internal.sql.prepped.merge.enabled")
+ .defaultValue(true)
+ .markAdvanced()
+ .sinceVersion("0.14.0")
+ .withDocumentation("Controls whether spark sql optimized merge is enabled.");
Review Comment:
Lets move this to HoodieWriteConfig and ignore the one we are adding in DataSourcEwriteOptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1641493001
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 585935c37efc35994dd721ba2d8f05c9cf775470 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669)
* 539cad2f3b8edde2211bd8ddeeb3feec15cd6e94 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268782851
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/SparkHoodieIndexFactory.java:
##########
@@ -44,8 +43,7 @@
*/
public final class SparkHoodieIndexFactory {
public static HoodieIndex createIndex(HoodieWriteConfig config) {
- boolean mergeIntoWrites = config.getProps().getBoolean(HoodieInternalConfig.SQL_MERGE_INTO_WRITES.key(),
- HoodieInternalConfig.SQL_MERGE_INTO_WRITES.defaultValue());
+ boolean mergeIntoWrites = config.getProps().getBoolean(HoodieWriteConfig.SPARK_SQL_MERGE_INTO_PREPPED_KEY, false);
Review Comment:
Replaced all instances of `mergeIntoWrites` to `sqlMergeIntoPrepped` (except for the original reference in `deduceWriterSchema`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1644714470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
}, {
"hash" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704",
"triggerID" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"triggerType" : "PUSH"
}, {
"hash" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18712",
"triggerID" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"triggerType" : "PUSH"
}, {
"hash" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18730",
"triggerID" : "1644022239",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0af9136d05a06e9a654f4c738d41a9063bdc1b00 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18712) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1644039151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
}, {
"hash" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704",
"triggerID" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"triggerType" : "PUSH"
}, {
"hash" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18712",
"triggerID" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"triggerType" : "PUSH"
}, {
"hash" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18730",
"triggerID" : "1644022239",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0af9136d05a06e9a654f4c738d41a9063bdc1b00 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18712) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18730)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268784995
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -617,6 +622,8 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
val hiveSyncConfig = buildHiveSyncConfig(sparkSession, hoodieCatalogTable, tableConfig)
+ val enableOptimizedMerge = sparkSession.sqlContext.conf.getConfString(SPARK_SQL_OPTIMIZED_WRITES.key(),
+ SPARK_SQL_OPTIMIZED_WRITES.defaultValue())
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1642985981
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
}, {
"hash" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704",
"triggerID" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbecc75443a72ac360f87b80913837c32d62c466 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691)
* cad8c2a2950628a8da9d0834735fbdb29871c85d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1267546687
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -642,11 +642,11 @@ object DataSourceWriteOptions {
val DROP_PARTITION_COLUMNS: ConfigProperty[java.lang.Boolean] = HoodieTableConfig.DROP_PARTITION_COLUMNS
val ENABLE_OPTIMIZED_SQL_WRITES: ConfigProperty[String] = ConfigProperty
- .key("hoodie.spark.sql.writes.optimized.enable")
+ .key("hoodie.spark.sql.optimized.writes.enable")
Review Comment:
generally we try to align the var naming to the key.
Lets name the variable
SPARK_SQL_OPTIMIZED_WRITES
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -721,6 +721,8 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final String WRITE_PREPPED_MERGE_KEY = "_hoodie.datasource.merge.into.prepped";
Review Comment:
can we add java docs calling out the purpose of this
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTableWithNonRecordKeyField.scala:
##########
@@ -22,122 +22,128 @@ import org.apache.hudi.{HoodieSparkUtils, ScalaAssertionSupport}
class TestMergeIntoTableWithNonRecordKeyField extends HoodieSparkSqlTestBase with ScalaAssertionSupport {
test("Test Merge into extra cond") {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts long
- |) using hudi
- | location '${tmp.getCanonicalPath}/$tableName'
- | tblproperties (
- | primaryKey ='id',
- | preCombineField = 'ts'
- | )
+ Seq(true, false).foreach { optimizedSqlEnabled =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey ='id',
+ | preCombineField = 'ts'
+ | )
""".stripMargin)
- val tableName2 = generateTableName
- spark.sql(
- s"""
- |create table $tableName2 (
- | id int,
- | name string,
- | price double,
- | ts long
- |) using hudi
- | location '${tmp.getCanonicalPath}/$tableName2'
- | tblproperties (
- | primaryKey ='id',
- | preCombineField = 'ts'
- | )
+ val tableName2 = generateTableName
+ spark.sql(
+ s"""
+ |create table $tableName2 (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | location '${tmp.getCanonicalPath}/$tableName2'
+ | tblproperties (
+ | primaryKey ='id',
+ | preCombineField = 'ts'
+ | )
""".stripMargin)
- spark.sql(
- s"""
- |insert into $tableName values
- | (1, 'a1', 10, 100),
- | (2, 'a2', 20, 200),
- | (3, 'a3', 20, 100)
- |""".stripMargin)
- spark.sql(
- s"""
- |insert into $tableName2 values
- | (1, 'u1', 10, 999),
- | (3, 'u3', 30, 9999),
- | (4, 'u4', 40, 99999)
- |""".stripMargin)
+ spark.sql(
+ s"""
+ |insert into $tableName values
+ | (1, 'a1', 10, 100),
+ | (2, 'a2', 20, 200),
+ | (3, 'a3', 20, 100)
+ |""".stripMargin)
+ spark.sql(
+ s"""
+ |insert into $tableName2 values
+ | (1, 'u1', 10, 999),
+ | (3, 'u3', 30, 9999),
+ | (4, 'u4', 40, 99999)
+ |""".stripMargin)
- spark.sql(
- s"""
- |merge into $tableName as oldData
- |using $tableName2
- |on oldData.id = $tableName2.id
- |when matched and oldData.price = $tableName2.price then update set oldData.name = $tableName2.name
- |
- |""".stripMargin)
+ // test with optimized sql merge enabled / disabled.
+ spark.sql(s"set hoodie.spark.sql.optimized.merge.enable=$optimizedSqlEnabled")
- checkAnswer(s"select id, name, price, ts from $tableName")(
- Seq(1, "u1", 10.0, 100),
- Seq(3, "a3", 20.0, 100),
- Seq(2, "a2", 20.0, 200)
- )
+ spark.sql(
+ s"""
+ |merge into $tableName as oldData
+ |using $tableName2
+ |on oldData.id = $tableName2.id
+ |when matched and oldData.price = $tableName2.price then update set oldData.name = $tableName2.name
+ |
+ |""".stripMargin)
- val errorMessage = if (HoodieSparkUtils.gteqSpark3_1) {
- "Only simple conditions of the form `t.id = s.id` using primary key or partition path " +
- "columns are allowed on tables with primary key. (illegal column(s) used: `price`"
- } else {
- "Only simple conditions of the form `t.id = s.id` using primary key or partition path " +
- "columns are allowed on tables with primary key. (illegal column(s) used: `price`;"
- }
+ checkAnswer(s"select id, name, price, ts from $tableName")(
+ Seq(1, "u1", 10.0, 100),
+ Seq(3, "a3", 20.0, 100),
+ Seq(2, "a2", 20.0, 200)
+ )
- checkException(
- s"""
- |merge into $tableName as oldData
- |using $tableName2
- |on oldData.id = $tableName2.id and oldData.price = $tableName2.price
- |when matched then update set oldData.name = $tableName2.name
- |when not matched then insert *
- |""".stripMargin)(errorMessage)
+ val errorMessage = if (HoodieSparkUtils.gteqSpark3_1) {
+ "Only simple conditions of the form `t.id = s.id` using primary key or partition path " +
+ "columns are allowed on tables with primary key. (illegal column(s) used: `price`"
+ } else {
+ "Only simple conditions of the form `t.id = s.id` using primary key or partition path " +
+ "columns are allowed on tables with primary key. (illegal column(s) used: `price`;"
+ }
- //test with multiple pks
- val tableName3 = generateTableName
- spark.sql(
- s"""
- |create table $tableName3 (
- | id int,
- | name string,
- | price double,
- | ts long
- |) using hudi
- | location '${tmp.getCanonicalPath}/$tableName3'
- | tblproperties (
- | primaryKey ='id,name',
- | preCombineField = 'ts'
- | )
+ checkException(
+ s"""
+ |merge into $tableName as oldData
+ |using $tableName2
+ |on oldData.id = $tableName2.id and oldData.price = $tableName2.price
+ |when matched then update set oldData.name = $tableName2.name
+ |when not matched then insert *
+ |""".stripMargin)(errorMessage)
+
+ //test with multiple pks
+ val tableName3 = generateTableName
+ spark.sql(
+ s"""
+ |create table $tableName3 (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ |) using hudi
+ | location '${tmp.getCanonicalPath}/$tableName3'
+ | tblproperties (
+ | primaryKey ='id,name',
+ | preCombineField = 'ts'
+ | )
""".stripMargin)
- val errorMessage2 = if (HoodieSparkUtils.gteqSpark3_1) {
- "Hudi tables with primary key are required to match on all primary key colums. Column: 'name' not found"
- } else {
- "Hudi tables with primary key are required to match on all primary key colums. Column: 'name' not found;"
- }
+ val errorMessage2 = if (HoodieSparkUtils.gteqSpark3_1) {
+ "Hudi tables with primary key are required to match on all primary key colums. Column: 'name' not found"
+ } else {
+ "Hudi tables with primary key are required to match on all primary key colums. Column: 'name' not found;"
+ }
- checkException(
- s"""
- |merge into $tableName3 as oldData
- |using $tableName2
- |on oldData.id = $tableName2.id
- |when matched then update set oldData.name = $tableName2.name
- |when not matched then insert *
- |""".stripMargin)(errorMessage2)
+ checkException(
+ s"""
+ |merge into $tableName3 as oldData
+ |using $tableName2
+ |on oldData.id = $tableName2.id
+ |when matched then update set oldData.name = $tableName2.name
+ |when not matched then insert *
+ |""".stripMargin)(errorMessage2)
+ }
}
}
test("Test pkless complex merge cond") {
withRecordType()(withTempDir { tmp =>
spark.sql("set hoodie.payload.combined.schema.validate = true")
+ spark.sql("set hoodie.spark.sql.optimized.merge.enable=true")
Review Comment:
don't we need to fix all these?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -146,9 +144,7 @@ class DefaultSource extends RelationProvider
mode: SaveMode,
optParams: Map[String, String],
rawDf: DataFrame): BaseRelation = {
- val df = if (optParams.getOrDefault(DATASOURCE_WRITE_PREPPED_KEY,
- optParams.getOrDefault(SQL_MERGE_INTO_WRITES.key(), SQL_MERGE_INTO_WRITES.defaultValue().toString))
- .equalsIgnoreCase("true")) {
+ val df = if (optParams.getOrDefault(DATASOURCE_WRITE_PREPPED_KEY, "false").toBoolean || optParams.getOrDefault(WRITE_PREPPED_MERGE_KEY, "false").toBoolean) {
Review Comment:
and the config key be
_hoodie.spark.sql.writes.prepped
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -721,6 +721,8 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final String WRITE_PREPPED_MERGE_KEY = "_hoodie.datasource.merge.into.prepped";
Review Comment:
lets name the variable
SPARK_SQL_MERGE_INTO_PREPPED
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -146,9 +144,7 @@ class DefaultSource extends RelationProvider
mode: SaveMode,
optParams: Map[String, String],
rawDf: DataFrame): BaseRelation = {
- val df = if (optParams.getOrDefault(DATASOURCE_WRITE_PREPPED_KEY,
- optParams.getOrDefault(SQL_MERGE_INTO_WRITES.key(), SQL_MERGE_INTO_WRITES.defaultValue().toString))
- .equalsIgnoreCase("true")) {
+ val df = if (optParams.getOrDefault(DATASOURCE_WRITE_PREPPED_KEY, "false").toBoolean || optParams.getOrDefault(WRITE_PREPPED_MERGE_KEY, "false").toBoolean) {
Review Comment:
lets also rename the variable
DATASOURCE_WRITE_PREPPED_KEY to
SPARK_SQL_WRITE_PREPPED_KEY
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -721,6 +721,8 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final String WRITE_PREPPED_MERGE_KEY = "_hoodie.datasource.merge.into.prepped";
Review Comment:
sorry. lets name this
_hoodie.spark.sql.merge.into.prepped
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1267607979
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -642,11 +642,11 @@ object DataSourceWriteOptions {
val DROP_PARTITION_COLUMNS: ConfigProperty[java.lang.Boolean] = HoodieTableConfig.DROP_PARTITION_COLUMNS
val ENABLE_OPTIMIZED_SQL_WRITES: ConfigProperty[String] = ConfigProperty
- .key("hoodie.spark.sql.writes.optimized.enable")
+ .key("hoodie.spark.sql.optimized.writes.enable")
Review Comment:
Fixed.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -146,9 +144,7 @@ class DefaultSource extends RelationProvider
mode: SaveMode,
optParams: Map[String, String],
rawDf: DataFrame): BaseRelation = {
- val df = if (optParams.getOrDefault(DATASOURCE_WRITE_PREPPED_KEY,
- optParams.getOrDefault(SQL_MERGE_INTO_WRITES.key(), SQL_MERGE_INTO_WRITES.defaultValue().toString))
- .equalsIgnoreCase("true")) {
+ val df = if (optParams.getOrDefault(DATASOURCE_WRITE_PREPPED_KEY, "false").toBoolean || optParams.getOrDefault(WRITE_PREPPED_MERGE_KEY, "false").toBoolean) {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1642648608
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* cbecc75443a72ac360f87b80913837c32d62c466 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1267531685
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -721,6 +721,8 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final String WRITE_PREPPED_MERGE_KEY = "_hoodie.datasource.merge.prepped";
+
Review Comment:
Fixed.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -642,11 +642,18 @@ object DataSourceWriteOptions {
val DROP_PARTITION_COLUMNS: ConfigProperty[java.lang.Boolean] = HoodieTableConfig.DROP_PARTITION_COLUMNS
val ENABLE_OPTIMIZED_SQL_WRITES: ConfigProperty[String] = ConfigProperty
- .key("hoodie.spark.sql.writes.optimized.enable")
+ .key("hoodie.spark.sql.optimized.writes.enable")
.defaultValue("true")
.markAdvanced()
.sinceVersion("0.14.0")
- .withDocumentation("Controls whether spark sql optimized update is enabled.")
+ .withDocumentation("Controls whether spark sql prepped update and delete is enabled.")
+
+ val ENABLE_OPTIMIZED_SQL_MERGE_WRITES: ConfigProperty[String] = ConfigProperty
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9203:
URL: https://github.com/apache/hudi/pull/9203#discussion_r1268910357
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala:
##########
@@ -648,7 +662,7 @@ case class MergeIntoHoodieTableCommand(mergeInto: MergeIntoTable) extends Hoodie
RECONCILE_SCHEMA.key -> "false",
CANONICALIZE_NULLABLE.key -> "false",
SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.key -> "true",
- HoodieInternalConfig.SQL_MERGE_INTO_WRITES.key -> "true",
+ HoodieWriteConfig.SPARK_SQL_MERGE_INTO_PREPPED_KEY -> enableOptimizedMerge,
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9203: [HUDI-6315] Feature flag for disabling prepped merge.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9203:
URL: https://github.com/apache/hudi/pull/9203#issuecomment-1643163767
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"triggerType" : "PUSH"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18667",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18665",
"triggerID" : "1640349773",
"triggerType" : "MANUAL"
}, {
"hash" : "585935c37efc35994dd721ba2d8f05c9cf775470",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18669",
"triggerID" : "1640483825",
"triggerType" : "MANUAL"
}, {
"hash" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18679",
"triggerID" : "539cad2f3b8edde2211bd8ddeeb3feec15cd6e94",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681",
"triggerID" : "cbecc75443a72ac360f87b80913837c32d62c466",
"triggerType" : "PUSH"
}, {
"hash" : "cbecc75443a72ac360f87b80913837c32d62c466",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691",
"triggerID" : "1642359770",
"triggerType" : "MANUAL"
}, {
"hash" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704",
"triggerID" : "cad8c2a2950628a8da9d0834735fbdb29871c85d",
"triggerType" : "PUSH"
}, {
"hash" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0af9136d05a06e9a654f4c738d41a9063bdc1b00",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbecc75443a72ac360f87b80913837c32d62c466 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18681) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18691)
* cad8c2a2950628a8da9d0834735fbdb29871c85d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18704)
* 0af9136d05a06e9a654f4c738d41a9063bdc1b00 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org