You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Zihan Li (Jira)" <ji...@apache.org> on 2024/03/25 18:23:00 UTC

[jira] [Created] (GOBBLIN-2024) Improve the performance of GobblinMCEWriter

Zihan Li created GOBBLIN-2024:
---------------------------------

             Summary: Improve the performance of GobblinMCEWriter
                 Key: GOBBLIN-2024
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-2024
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: Zihan Li


We are seeing slowness in the GobblinMCEWriter, which will cause us huge backlog when handling large topic and affect the other topics in the same container for GMIP. After benchmark, we identify that we can improve the computeSpec part to reduce the duplicate work, also we can add the file size information in GMCE to avoid calling HDFS for each file. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)