You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Adar Dembo (Jira)" <ji...@apache.org> on 2020/02/14 05:47:00 UTC
[jira] [Commented] (KUDU-3054) Init kudu.write_duration accumulator
lazily
[ https://issues.apache.org/jira/browse/KUDU-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036702#comment-17036702 ]
Adar Dembo commented on KUDU-3054:
----------------------------------
[~liupengcheng] thanks for filing this.
I can't load the images behind the links in your bug report. Could you add them directly as file attachments? Barring that, could you host them on another site, like imgur.com?
> Init kudu.write_duration accumulator lazily
> -------------------------------------------
>
> Key: KUDU-3054
> URL: https://issues.apache.org/jira/browse/KUDU-3054
> Project: Kudu
> Issue Type: Improvement
> Components: spark
> Affects Versions: 1.9.0
> Reporter: liupengcheng
> Priority: Major
>
> Currently, we encountered a issue in kudu-spark that will causing spark sql query failure:
> ```
> Job aborted due to stage failure: Total size of serialized results of 942 tasks (2.0 GB) is bigger than spark.driver.maxResultSize (2.0 GB)
> ```
> After carefully debug, we find out that it's the kudu.write_duration accumulators causing single spark task larger than 2M, thus all tasks size of the stage will bigger than the limit.
> However, this stage is just reading kudu table and do shuffle exchange, no writing any kudu tables.
> So I think should init this accumulator lazily in KuduContext to avoid such issues.
> !https://phabricator.d.xiaomi.net/file/data/fnnfiseg2cy4pzs4rlj7/PHID-FILE-6tj4q4w4hsvtba2xiaew/image.png!
>
> !https://phabricator.d.xiaomi.net/file/data/rf4vosfntshsqf645iwi/PHID-FILE-oaion5kpfutxoxjtjzrs/image.png!
> !https://phabricator.d.xiaomi.net/file/data/7cirhhktqarpxsdyisek/PHID-FILE-drhuwotg3dr7hnmocdex/image.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)