You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2025/02/22 01:36:21 UTC
[PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
nsivabalan opened a new pull request, #12874:
URL: https://github.com/apache/hudi/pull/12874
### Change Logs
- Fixing num spark tasks for clean action.
Validated the fix using simple quick start.
```
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.common.table.HoodieTableConfig._
import org.apache.hudi.config.HoodieWriteConfig._
import org.apache.hudi.keygen.constant.KeyGeneratorOptions._
import org.apache.hudi.common.model.HoodieRecord
import spark.implicits._
val tableName = "trips_table1"
val basePath = "file:///tmp/trips_table1"
val columns = Seq("ts","uuid","rider","driver","fare","city")
val data =
Seq((1695159649087L,"334e26e9-8355-45cc-97c6-c31daf0df330","rider-A","driver-K",19.10,"san_francisco"),
(1695091554788L,"e96c4396-3fad-413a-a942-4cb36106d721","rider-C","driver-M",27.70 ,"san_francisco"),
(1695046462179L,"9909a8b1-2d15-4d3d-8ec9-efc48c536a00","rider-D","driver-L",33.90 ,"san_francisco"),
(1695516137016L,"e3cf430c-889d-4015-bc98-59bdce1e530c","rider-F","driver-P",34.15,"sao_paulo" ),
(1695115999911L,"c8abbe79-8d89-47ea-b4ce-4d224bae5bfa","rider-J","driver-T",17.85,"chennai"));
var inserts = spark.createDataFrame(data).toDF(columns:_*)
inserts.write.format("hudi").
option("hoodie.datasource.write.partitionpath.field", "city").
option("hoodie.datasource.write.recordkey.field","uuid").
option("hoodie.table.name", tableName).
option("hoodie.datasource.write.operation","INSERT").
mode(Overwrite).
save(basePath)
val df1 = inserts.limit(1)
for (i <- 1 to 10) {
df1.write.format("hudi").
option("hoodie.datasource.write.partitionpath.field", "city").
option("hoodie.datasource.write.recordkey.field","uuid").
option("hoodie.datasource.write.operation", "INSERT").
mode(Append).
save(basePath)
}
```
Validated from spark UI
Before fix:
<img width="1770" alt="image" src="https://github.com/user-attachments/assets/e532460b-c8b5-4453-b2bd-60767db11444" />
After fix:
<img width="1774" alt="image" src="https://github.com/user-attachments/assets/dbbb8657-12fc-487b-842e-93bbbd4aa35a" />
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2699682529
CI is flaky and failures are irrelevant. Merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2676295669
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
}, {
"hash" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833",
"triggerID" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c73cd628b8b20e43cf77c43a71a95db74f4e71b7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2675924167
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 87a66056b190341981201bf9eee119725e426647 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2676235016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 87a66056b190341981201bf9eee119725e426647 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824)
* 7e3a6a4ad071745978b9138ced3b0b6f0e885a4e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12874:
URL: https://github.com/apache/hudi/pull/12874#discussion_r1966456842
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -148,6 +148,7 @@ public <I, K, V> Stream<ImmutablePair<K, V>> mapPartitionsToPairAndReduceByKey(
.map(e -> new Tuple2<>(e.getKey(), e.getValue())).iterator()
)
.reduceByKey(reduceFunc::apply)
+ .coalesce(parallelism)
Review Comment:
Should we also revisit other API implementations in `HoodieSparkEngineContext` for applying parallelism for the shuffling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12874:
URL: https://github.com/apache/hudi/pull/12874#discussion_r1966456190
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -148,6 +148,7 @@ public <I, K, V> Stream<ImmutablePair<K, V>> mapPartitionsToPairAndReduceByKey(
.map(e -> new Tuple2<>(e.getKey(), e.getValue())).iterator()
)
.reduceByKey(reduceFunc::apply)
+ .coalesce(parallelism)
Review Comment:
Is the intention to change the parallelism of `reduceByKey`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #12874:
URL: https://github.com/apache/hudi/pull/12874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12874:
URL: https://github.com/apache/hudi/pull/12874#discussion_r1980423227
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -148,6 +148,7 @@ public <I, K, V> Stream<ImmutablePair<K, V>> mapPartitionsToPairAndReduceByKey(
.map(e -> new Tuple2<>(e.getKey(), e.getValue())).iterator()
)
.reduceByKey(reduceFunc::apply)
+ .coalesce(parallelism)
Review Comment:
HUDI-9111 to follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2675923198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 87a66056b190341981201bf9eee119725e426647 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2676235502
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
}, {
"hash" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 87a66056b190341981201bf9eee119725e426647 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824)
* 7e3a6a4ad071745978b9138ced3b0b6f0e885a4e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832)
* c73cd628b8b20e43cf77c43a71a95db74f4e71b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2676234584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 87a66056b190341981201bf9eee119725e426647 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824)
* 7e3a6a4ad071745978b9138ced3b0b6f0e885a4e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2675980684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 87a66056b190341981201bf9eee119725e426647 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2699685716
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "747270a6d255199e6e3bda5e07246a3bf977adf9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "747270a6d255199e6e3bda5e07246a3bf977adf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 747270a6d255199e6e3bda5e07246a3bf977adf9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2676261538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
}, {
"hash" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833",
"triggerID" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e3a6a4ad071745978b9138ced3b0b6f0e885a4e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832)
* c73cd628b8b20e43cf77c43a71a95db74f4e71b7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2676235977
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
}, {
"hash" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e3a6a4ad071745978b9138ced3b0b6f0e885a4e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832)
* c73cd628b8b20e43cf77c43a71a95db74f4e71b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2699406557
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
}, {
"hash" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833",
"triggerID" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"triggerType" : "PUSH"
}, {
"hash" : "747270a6d255199e6e3bda5e07246a3bf977adf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4055",
"triggerID" : "747270a6d255199e6e3bda5e07246a3bf977adf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 747270a6d255199e6e3bda5e07246a3bf977adf9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4055)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2699234006
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
}, {
"hash" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833",
"triggerID" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"triggerType" : "PUSH"
}, {
"hash" : "747270a6d255199e6e3bda5e07246a3bf977adf9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4055",
"triggerID" : "747270a6d255199e6e3bda5e07246a3bf977adf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c73cd628b8b20e43cf77c43a71a95db74f4e71b7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833)
* 747270a6d255199e6e3bda5e07246a3bf977adf9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4055)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12874:
URL: https://github.com/apache/hudi/pull/12874#issuecomment-2699228062
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "87a66056b190341981201bf9eee119725e426647",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3824",
"triggerID" : "87a66056b190341981201bf9eee119725e426647",
"triggerType" : "PUSH"
}, {
"hash" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3832",
"triggerID" : "7e3a6a4ad071745978b9138ced3b0b6f0e885a4e",
"triggerType" : "PUSH"
}, {
"hash" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833",
"triggerID" : "c73cd628b8b20e43cf77c43a71a95db74f4e71b7",
"triggerType" : "PUSH"
}, {
"hash" : "747270a6d255199e6e3bda5e07246a3bf977adf9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "747270a6d255199e6e3bda5e07246a3bf977adf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c73cd628b8b20e43cf77c43a71a95db74f4e71b7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3833)
* 747270a6d255199e6e3bda5e07246a3bf977adf9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #12874:
URL: https://github.com/apache/hudi/pull/12874#discussion_r1966455425
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/common/TestHoodieSparkEngineContext.java:
##########
@@ -67,4 +78,49 @@ void testSetJobStatus() {
// Assert jobGroupId does not change
assertEquals(jobGroupId, context.getJavaSparkContext().getLocalProperty("spark.jobGroup.id"));
}
+
+ @Test
+ void testMapPartitionsToPairAndReduceByKey() {
+ int numPartitions = 6;
+ HoodieData<Integer> rddData = context.parallelize(
+ IntStream.rangeClosed(0, 99).boxed().collect(Collectors.toList()), numPartitions);
+
+ //
+ /* output from map to pair.
+ 1 = 0, 1
+ 2 = 1, 5
+ 3 = 1, 5
+ 4 = 2, 10
+ 5 = 2, 10
+ 6 = 3, 15
+ 7 = 3, 15
+ 8 = 4, 20
+
+ And then we do reduce by key, where we sum up the values.
+ */
+ Stream<ImmutablePair<Integer, Integer>> result = context.mapPartitionsToPairAndReduceByKey(rddData.collectAsList().stream(),
+ new SerializablePairFlatMapTestFunc(), new SerializableReduceTestFunc(), 6);
+ result.forEach(entry -> {
+ assertEquals(entry.getKey() * 10, entry.getValue());
+ });
Review Comment:
Also assert the number of items in `result`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9067] Fixing num spark tasks for clean action [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12874:
URL: https://github.com/apache/hudi/pull/12874#discussion_r1966523777
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -148,6 +148,7 @@ public <I, K, V> Stream<ImmutablePair<K, V>> mapPartitionsToPairAndReduceByKey(
.map(e -> new Tuple2<>(e.getKey(), e.getValue())).iterator()
)
.reduceByKey(reduceFunc::apply)
+ .coalesce(parallelism)
Review Comment:
yes, btw, found that reduceKyKey itslef takes an additional arg. I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org