You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Jingsong Lee (Jira)" <ji...@apache.org> on 2019/11/12 11:04:00 UTC

[jira] [Created] (FLINK-14722) Optimize mapred.HadoopInputSplit to not serialize conf when split is not configurable

Jingsong Lee created FLINK-14722:
------------------------------------

             Summary: Optimize mapred.HadoopInputSplit to not serialize conf when split is not configurable
                 Key: FLINK-14722
                 URL: https://issues.apache.org/jira/browse/FLINK-14722
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Hadoop Compatibility
            Reporter: Jingsong Lee
             Fix For: 1.10.0


JobConf may very big, contains hundreds of configurations, if it is serialized by every split, that will significantly reduce performance.

Consider thousands of splits, the akka thread of JobMaster will all on the serialization of conf. That may will lead to various akka timeouts too.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)