You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2023/03/26 03:29:00 UTC
[jira] [Updated] (HUDI-5238) Hudi throwing "PipeBroken" exception during Merging on GCS
[ https://issues.apache.org/jira/browse/HUDI-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymond Xu updated HUDI-5238:
-----------------------------
Sprint: 2022/11/15, 2022/11/29, 2022/12/12, 0.13.0 Final Sprint, 0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3, Sprint 2023-01-31, Sprint 2023-02-14, Sprint 2023-02-28 (was: 2022/11/15, 2022/11/29, 2022/12/12, 0.13.0 Final Sprint, 0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3, Sprint 2023-01-31, Sprint 2023-02-14)
> Hudi throwing "PipeBroken" exception during Merging on GCS
> ----------------------------------------------------------
>
> Key: HUDI-5238
> URL: https://issues.apache.org/jira/browse/HUDI-5238
> Project: Apache Hudi
> Issue Type: Bug
> Affects Versions: 0.12.1
> Reporter: Alexey Kudinkin
> Assignee: Alexey Kudinkin
> Priority: Critical
> Labels: pull-request-available
> Fix For: 0.14.0
>
>
> Originally reported at [https://github.com/apache/hudi/issues/7234]
> -------
>
> Root-cause:
> Basically, the reason it’s failing is following: # GCS uses PipeInputStream/PipeOutputStream comprising reading/writing ends of the “pipe” it’s using for unidirectional comm b/w Threads
> # PipeInputStream (for whatever reason) remembers the thread that actually wrote into the pipe
> # In BoundedInMemoryQueue we’re bootstrapping new executors (read, threads) for reading and _writing_ (it’s only used in HoodieMergeHandle, and in bulk-insert)
> # When we’re done writing in HoodieMergeHelper, we’re shutting down *first* BIMQ, then the HoodieMergeHandle, and that’s exactly the reason why it’s failing
>
> Issue has been introduced at [https://github.com/apache/hudi/pull/4264/files]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)