You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "fhan688 (via GitHub)" <gi...@apache.org> on 2024/12/18 13:07:42 UTC
[PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
fhan688 opened a new pull request, #12518:
URL: https://github.com/apache/hudi/pull/12518
### Change Logs
`close` is slow when large amount rowCreateHandles in BulkInsertWriterHelper, this PR close this handles in parallel.
### Impact
hudi-flink-datasource
### Risk level (write none, low medium or high below)
Low
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2572238988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
}, {
"hash" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e32ed77e587ecae84d8c33fa4647381961c378e0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618)
* a0d49d11bbea8c0f04e1e1ee64841dca3b79916f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2574509932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
}, {
"hash" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651",
"triggerID" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"triggerType" : "PUSH"
}, {
"hash" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2680",
"triggerID" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"triggerType" : "PUSH"
}, {
"hash" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2684",
"triggerID" : "2574508378",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2680) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2684)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1903782621
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,28 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ if (handles.isEmpty()) {
+ return;
+ }
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
Review Comment:
yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12518:
URL: https://github.com/apache/hudi/pull/12518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1904824818
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,28 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ if (handles.isEmpty()) {
+ return;
+ }
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
Review Comment:
Let's hard code a 100 max parallelism now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2574338788
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
}, {
"hash" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651",
"triggerID" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"triggerType" : "PUSH"
}, {
"hash" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0d49d11bbea8c0f04e1e1ee64841dca3b79916f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651)
* f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2551281519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7c244b90bf40cd65060abc79e3214a90d676830a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2553382736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7c244b90bf40cd65060abc79e3214a90d676830a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493)
* eb73f0006086cdb2a4820a59aa99d830313e2ca6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1900512219
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,25 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
+ allOf(handles.values().stream()
+ .map(rowCreateHandle -> CompletableFuture.supplyAsync(() -> {
+ try {
+ LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
+ return rowCreateHandle.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("IOE during rowCreateHandle.close()", e);
+ }
+ }, executorService))
+ .collect(Collectors.toList())
+ ).whenComplete((result, throwable) -> {
+ writeStatusList.addAll(result);
+ }).join();
+ try {
+ executorService.shutdown();
+ executorService.awaitTermination(24, TimeUnit.DAYS);
Review Comment:
Let's set it up as 10 minutes first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2553379662
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7c244b90bf40cd65060abc79e3214a90d676830a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493)
* eb73f0006086cdb2a4820a59aa99d830313e2ca6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2567483076
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e32ed77e587ecae84d8c33fa4647381961c378e0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2567404877
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb73f0006086cdb2a4820a59aa99d830313e2ca6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515)
* e32ed77e587ecae84d8c33fa4647381961c378e0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1891583674
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,25 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
+ allOf(handles.values().stream()
+ .map(rowCreateHandle -> CompletableFuture.supplyAsync(() -> {
+ try {
+ LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
+ return rowCreateHandle.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("IOE during rowCreateHandle.close()", e);
+ }
+ }, executorService))
+ .collect(Collectors.toList())
+ ).whenComplete((result, throwable) -> {
+ writeStatusList.addAll(result);
+ }).join();
+ try {
+ executorService.shutdown();
+ executorService.awaitTermination(24, TimeUnit.DAYS);
Review Comment:
Sorry! 24 hours. this indicates a long timeout , aiming to waiting for all handles closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1899354718
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,25 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
+ allOf(handles.values().stream()
+ .map(rowCreateHandle -> CompletableFuture.supplyAsync(() -> {
+ try {
+ LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
+ return rowCreateHandle.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("IOE during rowCreateHandle.close()", e);
+ }
+ }, executorService))
+ .collect(Collectors.toList())
+ ).whenComplete((result, throwable) -> {
+ writeStatusList.addAll(result);
+ }).join();
+ try {
+ executorService.shutdown();
+ executorService.awaitTermination(24, TimeUnit.DAYS);
Review Comment:
Teacher Danny, PTAL @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1904824818
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,28 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ if (handles.isEmpty()) {
+ return;
+ }
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
Review Comment:
Let's hard code a 10 max parallelism now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2572304864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
}, {
"hash" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651",
"triggerID" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0d49d11bbea8c0f04e1e1ee64841dca3b79916f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2572240161
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
}, {
"hash" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651",
"triggerID" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e32ed77e587ecae84d8c33fa4647381961c378e0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618)
* a0d49d11bbea8c0f04e1e1ee64841dca3b79916f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1903691240
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,28 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ if (handles.isEmpty()) {
+ return;
+ }
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
Review Comment:
We may need to control the parallelism of this `newFixedThreadPool` to prevent huge concurrency when there are many handles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1897081357
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,25 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
+ allOf(handles.values().stream()
+ .map(rowCreateHandle -> CompletableFuture.supplyAsync(() -> {
+ try {
+ LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
+ return rowCreateHandle.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("IOE during rowCreateHandle.close()", e);
+ }
+ }, executorService))
+ .collect(Collectors.toList())
+ ).whenComplete((result, throwable) -> {
+ writeStatusList.addAll(result);
+ }).join();
+ try {
+ executorService.shutdown();
+ executorService.awaitTermination(24, TimeUnit.DAYS);
Review Comment:
Did you estimate the normal time cost for the close? Maybe just 1 minute should be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2567403336
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb73f0006086cdb2a4820a59aa99d830313e2ca6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515)
* e32ed77e587ecae84d8c33fa4647381961c378e0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1893450373
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,25 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
+ allOf(handles.values().stream()
+ .map(rowCreateHandle -> CompletableFuture.supplyAsync(() -> {
+ try {
+ LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
+ return rowCreateHandle.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("IOE during rowCreateHandle.close()", e);
+ }
+ }, executorService))
+ .collect(Collectors.toList())
+ ).whenComplete((result, throwable) -> {
+ writeStatusList.addAll(result);
+ }).join();
+ try {
+ executorService.shutdown();
+ executorService.awaitTermination(24, TimeUnit.DAYS);
Review Comment:
Still too long, the close should be fast right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2551285322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7c244b90bf40cd65060abc79e3214a90d676830a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2553514088
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eb73f0006086cdb2a4820a59aa99d830313e2ca6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1891057720
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,25 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
+ allOf(handles.values().stream()
+ .map(rowCreateHandle -> CompletableFuture.supplyAsync(() -> {
+ try {
+ LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
+ return rowCreateHandle.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("IOE during rowCreateHandle.close()", e);
+ }
+ }, executorService))
+ .collect(Collectors.toList())
+ ).whenComplete((result, throwable) -> {
+ writeStatusList.addAll(result);
+ }).join();
+ try {
+ executorService.shutdown();
+ executorService.awaitTermination(24, TimeUnit.DAYS);
Review Comment:
24 days?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1896620125
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,25 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
+ allOf(handles.values().stream()
+ .map(rowCreateHandle -> CompletableFuture.supplyAsync(() -> {
+ try {
+ LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
+ return rowCreateHandle.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("IOE during rowCreateHandle.close()", e);
+ }
+ }, executorService))
+ .collect(Collectors.toList())
+ ).whenComplete((result, throwable) -> {
+ writeStatusList.addAll(result);
+ }).join();
+ try {
+ executorService.shutdown();
+ executorService.awaitTermination(24, TimeUnit.DAYS);
Review Comment:
> Still too long, the close should be fast right?
I agree, close should be fast. here awaitTermination is actually a timeout, 24 Hours is too long , how about wait for a Flink CP interval?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2568630228
@fhan688 Can you check the Azure CI failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1904113010
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,28 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ if (handles.isEmpty()) {
+ return;
+ }
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
Review Comment:
thanks, make sense. maybe we can introduce a config and set a default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2574406838
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
}, {
"hash" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651",
"triggerID" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"triggerType" : "PUSH"
}, {
"hash" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2680",
"triggerID" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2680)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2551453950
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7c244b90bf40cd65060abc79e3214a90d676830a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12518:
URL: https://github.com/apache/hudi/pull/12518#discussion_r1897230023
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriterHelper.java:
##########
@@ -157,9 +165,25 @@ private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throw
}
public void close() throws IOException {
- for (HoodieRowDataCreateHandle rowCreateHandle : handles.values()) {
- LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
- writeStatusList.add(closeWriteHandle(rowCreateHandle));
+ ExecutorService executorService = Executors.newFixedThreadPool(handles.size());
+ allOf(handles.values().stream()
+ .map(rowCreateHandle -> CompletableFuture.supplyAsync(() -> {
+ try {
+ LOG.info("Closing bulk insert file " + rowCreateHandle.getFileName());
+ return rowCreateHandle.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("IOE during rowCreateHandle.close()", e);
+ }
+ }, executorService))
+ .collect(Collectors.toList())
+ ).whenComplete((result, throwable) -> {
+ writeStatusList.addAll(result);
+ }).join();
+ try {
+ executorService.shutdown();
+ executorService.awaitTermination(24, TimeUnit.DAYS);
Review Comment:
1 minute maybe not enough when HDFS is busy. or we can make this a config and set default value 1 minute?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2574340004
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2493",
"triggerID" : "7c244b90bf40cd65060abc79e3214a90d676830a",
"triggerType" : "PUSH"
}, {
"hash" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2515",
"triggerID" : "eb73f0006086cdb2a4820a59aa99d830313e2ca6",
"triggerType" : "PUSH"
}, {
"hash" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2618",
"triggerID" : "e32ed77e587ecae84d8c33fa4647381961c378e0",
"triggerType" : "PUSH"
}, {
"hash" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651",
"triggerID" : "a0d49d11bbea8c0f04e1e1ee64841dca3b79916f",
"triggerType" : "PUSH"
}, {
"hash" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2680",
"triggerID" : "f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0d49d11bbea8c0f04e1e1ee64841dca3b79916f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2651)
* f2ec7962b2c67c563a8a3ff1ed6e2c2418c052c1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2680)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2574508378
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8782] BulkInsertWriterHelper parallel close [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12518:
URL: https://github.com/apache/hudi/pull/12518#issuecomment-2574580894
The Azure CI passed: https://dev.azure.com/apachehudi/hudi-oss-ci/_build/results?buildId=2680&view=logs&j=9273dbcb-a208-5cd0-5541-8b9925cb3da0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org