You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2024/03/07 19:39:23 UTC
[PR] [HUDI-7489] get rid of collect in row writer clustering [hudi]
jonvex opened a new pull request, #10836:
URL: https://github.com/apache/hudi/pull/10836
### Change Logs
Collect of all the write statuses will be a ton of data on the driver. We can avoid this because the data is parallelized again immediately after the collect
### Impact
Prevent OOM and speed up row writer clustering
### Risk level (write none, low medium or high below)
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #10836:
URL: https://github.com/apache/hudi/pull/10836#discussion_r1517073079
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -149,53 +150,53 @@ object HoodieDatasetBulkInsertHelper
arePartitionRecordsSorted: Boolean,
shouldPreserveHoodieMetadata: Boolean): HoodieData[WriteStatus] = {
val schema = dataset.schema
- val writeStatuses = injectSQLConf(dataset.queryExecution.toRdd.mapPartitions(iter => {
- val taskContextSupplier: TaskContextSupplier = table.getTaskContextSupplier
- val taskPartitionId = taskContextSupplier.getPartitionIdSupplier.get
- val taskId = taskContextSupplier.getStageIdSupplier.get.toLong
- val taskEpochId = taskContextSupplier.getAttemptIdSupplier.get
+ HoodieJavaRDD.of(
+ injectSQLConf(dataset.queryExecution.toRdd.mapPartitions(iter => {
+ val taskContextSupplier: TaskContextSupplier = table.getTaskContextSupplier
+ val taskPartitionId = taskContextSupplier.getPartitionIdSupplier.get
+ val taskId = taskContextSupplier.getStageIdSupplier.get.toLong
+ val taskEpochId = taskContextSupplier.getAttemptIdSupplier.get
- val writer = writeConfig.getIndexType match {
- case HoodieIndex.IndexType.BUCKET if writeConfig.getBucketIndexEngineType
- == BucketIndexEngineType.CONSISTENT_HASHING =>
- new ConsistentBucketBulkInsertDataInternalWriterHelper(
- table,
- writeConfig,
- instantTime,
- taskPartitionId,
- taskId,
- taskEpochId,
- schema,
- writeConfig.populateMetaFields,
- arePartitionRecordsSorted,
- shouldPreserveHoodieMetadata)
- case _ =>
- new BulkInsertDataInternalWriterHelper(
- table,
- writeConfig,
- instantTime,
- taskPartitionId,
- taskId,
- taskEpochId,
- schema,
- writeConfig.populateMetaFields,
- arePartitionRecordsSorted,
- shouldPreserveHoodieMetadata)
- }
+ val writer = writeConfig.getIndexType match {
+ case HoodieIndex.IndexType.BUCKET if writeConfig.getBucketIndexEngineType
+ == BucketIndexEngineType.CONSISTENT_HASHING =>
+ new ConsistentBucketBulkInsertDataInternalWriterHelper(
+ table,
+ writeConfig,
+ instantTime,
+ taskPartitionId,
+ taskId,
+ taskEpochId,
+ schema,
+ writeConfig.populateMetaFields,
+ arePartitionRecordsSorted,
+ shouldPreserveHoodieMetadata)
+ case _ =>
+ new BulkInsertDataInternalWriterHelper(
+ table,
+ writeConfig,
+ instantTime,
+ taskPartitionId,
+ taskId,
+ taskEpochId,
+ schema,
+ writeConfig.populateMetaFields,
+ arePartitionRecordsSorted,
+ shouldPreserveHoodieMetadata)
+ }
- try {
- iter.foreach(writer.write)
- } catch {
- case t: Throwable =>
- writer.abort()
- throw t
- } finally {
- writer.close()
- }
+ try {
+ iter.foreach(writer.write)
+ } catch {
+ case t: Throwable =>
+ writer.abort()
+ throw t
+ } finally {
+ writer.close()
+ }
- writer.getWriteStatuses.asScala.iterator
- }), SQLConf.get).collect()
Review Comment:
Nice catch ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1986281572
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1988459576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "1988444029",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1989047472
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
}, {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1988444029",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1988444029",
"triggerType" : "MANUAL"
}, {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1985103920
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984621241
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836",
"triggerID" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1985244800
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22843",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1989062879
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1988444029",
"triggerType" : "MANUAL"
}, {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22880",
"triggerID" : "1988444029",
"triggerType" : "MANUAL"
}, {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22880",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22880)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984928127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3a929e52bed04c7752a4540069614bd3d708d11 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984832640
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836",
"triggerID" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"triggerType" : "PUSH"
}, {
"hash" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836)
* f3a929e52bed04c7752a4540069614bd3d708d11 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1985109727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22843",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22843)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1988444490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1988444029",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984840258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836",
"triggerID" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"triggerType" : "PUSH"
}, {
"hash" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837",
"triggerID" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836)
* f3a929e52bed04c7752a4540069614bd3d708d11 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984933674
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837",
"triggerID" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3a929e52bed04c7752a4540069614bd3d708d11 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837)
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1986269221
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on code in PR #10836:
URL: https://github.com/apache/hudi/pull/10836#discussion_r1517113136
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/ConsistentBucketIndexUtils.java:
##########
@@ -210,7 +210,15 @@ private static void createCommitMarker(HoodieTable table, Path fileStatus, Path
if (fs.exists(fullPath)) {
return;
}
- FileIOUtils.createFileInPath(fs, fullPath, Option.of(getUTF8Bytes(StringUtils.EMPTY_STRING)));
+ //prevent exception from race condition. We are ok with the file being created in another thread, so we should
+ // check for the marker after catching the exception and we don't need to fail if the file exists
+ try {
+ FileIOUtils.createFileInPath(fs, fullPath, Option.of(getUTF8Bytes(StringUtils.EMPTY_STRING)));
+ } catch (HoodieIOException e) {
+ if (!fs.exists(fullPath)) {
+ throw e;
+ }
+ }
Review Comment:
@jonvex LGTM.
BTW, it's better if we could add some logger here if catch exception to help debug unexpected behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984938449
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837",
"triggerID" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3a929e52bed04c7752a4540069614bd3d708d11 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837)
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1986379651
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852)
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1986434178
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22852",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1988967615
azure passing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984889599
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836",
"triggerID" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"triggerType" : "PUSH"
}, {
"hash" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837",
"triggerID" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3a929e52bed04c7752a4540069614bd3d708d11 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984369583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836",
"triggerID" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22836)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984358872
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f92d4eb347ab8a2fcb1aeb5ea278c64e730089a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1985190415
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22843",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22843)
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1988444029
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1989188687
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
}, {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22880",
"triggerID" : "1988444029",
"triggerType" : "MANUAL"
}, {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22880",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1988444029",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22880)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on code in PR #10836:
URL: https://github.com/apache/hudi/pull/10836#discussion_r1517113136
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/ConsistentBucketIndexUtils.java:
##########
@@ -210,7 +210,15 @@ private static void createCommitMarker(HoodieTable table, Path fileStatus, Path
if (fs.exists(fullPath)) {
return;
}
- FileIOUtils.createFileInPath(fs, fullPath, Option.of(getUTF8Bytes(StringUtils.EMPTY_STRING)));
+ //prevent exception from race condition. We are ok with the file being created in another thread, so we should
+ // check for the marker after catching the exception and we don't need to fail if the file exists
+ try {
+ FileIOUtils.createFileInPath(fs, fullPath, Option.of(getUTF8Bytes(StringUtils.EMPTY_STRING)));
+ } catch (HoodieIOException e) {
+ if (!fs.exists(fullPath)) {
+ throw e;
+ }
+ }
Review Comment:
@jonvex LGTM.
BTW, it's better if we could add some logger when catch exception here to help debug unexpected behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10836:
URL: https://github.com/apache/hudi/pull/10836#discussion_r1517067137
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bucket/ConsistentBucketIndexUtils.java:
##########
@@ -210,7 +210,15 @@ private static void createCommitMarker(HoodieTable table, Path fileStatus, Path
if (fs.exists(fullPath)) {
return;
}
- FileIOUtils.createFileInPath(fs, fullPath, Option.of(getUTF8Bytes(StringUtils.EMPTY_STRING)));
+ //prevent exception from race condition. We are ok with the file being created in another thread, so we should
+ // check for the marker after catching the exception and we don't need to fail if the file exists
+ try {
+ FileIOUtils.createFileInPath(fs, fullPath, Option.of(getUTF8Bytes(StringUtils.EMPTY_STRING)));
+ } catch (HoodieIOException e) {
+ if (!fs.exists(fullPath)) {
+ throw e;
+ }
+ }
Review Comment:
@beyond1920 can you please take a look at this? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1984976158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22837",
"triggerID" : "f3a929e52bed04c7752a4540069614bd3d708d11",
"triggerType" : "PUSH"
}, {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22841)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1986212820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 743f394ba5d3b6f7ebe79d399fb8d11d50a26a3b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10836:
URL: https://github.com/apache/hudi/pull/10836#issuecomment-1989028734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "72a23b30a71d227e54ee63cf5684215fb3d2b2f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 72a23b30a71d227e54ee63cf5684215fb3d2b2f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #10836:
URL: https://github.com/apache/hudi/pull/10836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org