You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "KnightChess (via GitHub)" <gi...@apache.org> on 2024/06/01 08:15:07 UTC
[PR] [HUDI-5956] fix spark DAG ui when write [hudi]
KnightChess opened a new pull request, #11376:
URL: https://github.com/apache/hudi/pull/11376
### Change Logs
DataSet.ofRows will create new QueryExecution, which is not registered, so the dag metric will not update when running job
### Impact
None
### Risk level (write none, low medium or high below)
None
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5956] fix spark DAG ui when write [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess closed pull request #11376: [HUDI-5956] fix spark DAG ui when write
URL: https://github.com/apache/hudi/pull/11376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5956] fix spark DAG ui when write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11376:
URL: https://github.com/apache/hudi/pull/11376#issuecomment-2143373862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7d4e56addf91ad02647000894c11e6b404f86518",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24183",
"triggerID" : "7d4e56addf91ad02647000894c11e6b404f86518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d4e56addf91ad02647000894c11e6b404f86518 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24183)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5956] fix spark DAG ui when write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11376:
URL: https://github.com/apache/hudi/pull/11376#issuecomment-2143391741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7d4e56addf91ad02647000894c11e6b404f86518",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24183",
"triggerID" : "7d4e56addf91ad02647000894c11e6b404f86518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d4e56addf91ad02647000894c11e6b404f86518 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24183)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5956] fix spark DAG ui when write [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on code in PR #11376:
URL: https://github.com/apache/hudi/pull/11376#discussion_r1624306145
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -174,32 +174,43 @@ class HoodieSparkSqlWriterInternal {
sourceDf: DataFrame,
streamingWritesParamsOpt: Option[StreamingWriteParams] = Option.empty,
hoodieWriteClient: Option[SparkRDDWriteClient[_]] = Option.empty):
-
(Boolean, HOption[String], HOption[String], HOption[String], SparkRDDWriteClient[_], HoodieTableConfig) = {
- var succeeded = false
- var counter = 0
- val maxRetry: Integer = Integer.parseInt(optParams.getOrElse(HoodieWriteConfig.NUM_RETRIES_ON_CONFLICT_FAILURES.key(), HoodieWriteConfig.NUM_RETRIES_ON_CONFLICT_FAILURES.defaultValue().toString))
- var toReturn: (Boolean, HOption[String], HOption[String], HOption[String], SparkRDDWriteClient[_], HoodieTableConfig) = null
- while (counter <= maxRetry && !succeeded) {
- try {
- toReturn = writeInternal(sqlContext, mode, optParams, sourceDf, streamingWritesParamsOpt, hoodieWriteClient)
- if (counter > 0) {
- log.warn(s"Succeeded with attempt no $counter")
- }
- succeeded = true
- } catch {
- case e: HoodieWriteConflictException =>
- val writeConcurrencyMode = optParams.getOrElse(HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key(), HoodieWriteConfig.WRITE_CONCURRENCY_MODE.defaultValue())
- if (WriteConcurrencyMode.supportsMultiWriter(writeConcurrencyMode) && counter < maxRetry) {
- counter += 1
- log.warn(s"Conflict found. Retrying again for attempt no $counter")
- } else {
- throw e
+ val retryWrite: () => (Boolean, HOption[String], HOption[String], HOption[String], SparkRDDWriteClient[_], HoodieTableConfig) = () => {
+ var succeeded = false
+ var counter = 0
+ val maxRetry: Integer = Integer.parseInt(optParams.getOrElse(HoodieWriteConfig.NUM_RETRIES_ON_CONFLICT_FAILURES.key(), HoodieWriteConfig.NUM_RETRIES_ON_CONFLICT_FAILURES.defaultValue().toString))
+ var toReturn: (Boolean, HOption[String], HOption[String], HOption[String], SparkRDDWriteClient[_], HoodieTableConfig) = null
+
+ while (counter <= maxRetry && !succeeded) {
+ try {
+ toReturn = writeInternal(sqlContext, mode, optParams, sourceDf, streamingWritesParamsOpt, hoodieWriteClient)
+ if (counter > 0) {
+ log.warn(s"Succeeded with attempt no $counter")
}
+ succeeded = true
+ } catch {
+ case e: HoodieWriteConflictException =>
+ val writeConcurrencyMode = optParams.getOrElse(HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key(), HoodieWriteConfig.WRITE_CONCURRENCY_MODE.defaultValue())
+ if (WriteConcurrencyMode.supportsMultiWriter(writeConcurrencyMode) && counter < maxRetry) {
+ counter += 1
+ log.warn(s"Conflict found. Retrying again for attempt no $counter")
+ } else {
+ throw e
+ }
+ }
}
+ toReturn
+ }
+
+ val executionId = getExecutionId(sqlContext.sparkContext, sourceDf.queryExecution)
+ if (executionId.isEmpty) {
+ sparkAdapter.sqlExecutionWithNewExecutionId(sourceDf.sparkSession, sourceDf.queryExecution, Option("Hudi Command"))(
Review Comment:
this executionId will be sub-list in rootExecutionId after this pr https://github.com/apache/spark/pull/40403, so ignore this `TODO` https://github.com/apache/hudi/pull/8233#discussion_r1298071684, cc @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5956] fix spark DAG ui when write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11376:
URL: https://github.com/apache/hudi/pull/11376#issuecomment-2143371096
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7d4e56addf91ad02647000894c11e6b404f86518",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d4e56addf91ad02647000894c11e6b404f86518",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d4e56addf91ad02647000894c11e6b404f86518 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5956] fix spark DAG ui when write [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11376:
URL: https://github.com/apache/hudi/pull/11376#discussion_r1625210183
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -174,32 +174,43 @@ class HoodieSparkSqlWriterInternal {
sourceDf: DataFrame,
streamingWritesParamsOpt: Option[StreamingWriteParams] = Option.empty,
hoodieWriteClient: Option[SparkRDDWriteClient[_]] = Option.empty):
-
(Boolean, HOption[String], HOption[String], HOption[String], SparkRDDWriteClient[_], HoodieTableConfig) = {
- var succeeded = false
- var counter = 0
- val maxRetry: Integer = Integer.parseInt(optParams.getOrElse(HoodieWriteConfig.NUM_RETRIES_ON_CONFLICT_FAILURES.key(), HoodieWriteConfig.NUM_RETRIES_ON_CONFLICT_FAILURES.defaultValue().toString))
- var toReturn: (Boolean, HOption[String], HOption[String], HOption[String], SparkRDDWriteClient[_], HoodieTableConfig) = null
- while (counter <= maxRetry && !succeeded) {
- try {
- toReturn = writeInternal(sqlContext, mode, optParams, sourceDf, streamingWritesParamsOpt, hoodieWriteClient)
- if (counter > 0) {
- log.warn(s"Succeeded with attempt no $counter")
- }
- succeeded = true
- } catch {
- case e: HoodieWriteConflictException =>
- val writeConcurrencyMode = optParams.getOrElse(HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key(), HoodieWriteConfig.WRITE_CONCURRENCY_MODE.defaultValue())
- if (WriteConcurrencyMode.supportsMultiWriter(writeConcurrencyMode) && counter < maxRetry) {
- counter += 1
- log.warn(s"Conflict found. Retrying again for attempt no $counter")
- } else {
- throw e
+ val retryWrite: () => (Boolean, HOption[String], HOption[String], HOption[String], SparkRDDWriteClient[_], HoodieTableConfig) = () => {
+ var succeeded = false
+ var counter = 0
+ val maxRetry: Integer = Integer.parseInt(optParams.getOrElse(HoodieWriteConfig.NUM_RETRIES_ON_CONFLICT_FAILURES.key(), HoodieWriteConfig.NUM_RETRIES_ON_CONFLICT_FAILURES.defaultValue().toString))
+ var toReturn: (Boolean, HOption[String], HOption[String], HOption[String], SparkRDDWriteClient[_], HoodieTableConfig) = null
+
+ while (counter <= maxRetry && !succeeded) {
+ try {
+ toReturn = writeInternal(sqlContext, mode, optParams, sourceDf, streamingWritesParamsOpt, hoodieWriteClient)
+ if (counter > 0) {
+ log.warn(s"Succeeded with attempt no $counter")
}
+ succeeded = true
+ } catch {
+ case e: HoodieWriteConflictException =>
+ val writeConcurrencyMode = optParams.getOrElse(HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key(), HoodieWriteConfig.WRITE_CONCURRENCY_MODE.defaultValue())
+ if (WriteConcurrencyMode.supportsMultiWriter(writeConcurrencyMode) && counter < maxRetry) {
+ counter += 1
+ log.warn(s"Conflict found. Retrying again for attempt no $counter")
+ } else {
+ throw e
+ }
+ }
}
+ toReturn
+ }
+
+ val executionId = getExecutionId(sqlContext.sparkContext, sourceDf.queryExecution)
+ if (executionId.isEmpty) {
+ sparkAdapter.sqlExecutionWithNewExecutionId(sourceDf.sparkSession, sourceDf.queryExecution, Option("Hudi Command"))(
Review Comment:
@jonvex do you have intreast for the review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org