You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "CruiseCheung (Jira)" <ji...@apache.org> on 2021/04/06 13:23:00 UTC
[jira] [Updated] (TEZ-4301) TezChlid OOM cause PipelineSorter may
create too many sort span
[ https://issues.apache.org/jira/browse/TEZ-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
CruiseCheung updated TEZ-4301:
------------------------------
Attachment: (was: TEZ-4301.1.patch)
> TezChlid OOM cause PipelineSorter may create too many sort span
> ---------------------------------------------------------------
>
> Key: TEZ-4301
> URL: https://issues.apache.org/jira/browse/TEZ-4301
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2
> Environment:
> Reporter: CruiseCheung
> Priority: Minor
> Attachments: TEZ-4301.1.patch
>
>
> When previous span perItem is much large than others, reserved.remaining() is large。This may create too many min sort span，cause now we use perItem length to deduce next sort span's metaSize. In production, we may have 1 million sort span !
>
> Previous sort span:
> [TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=965738496, reserved.metasize=16777216
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem = 1400122
> [TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0, length=677, time=0
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676> (span=Span[16777216,947882594]),
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem = 442
> [TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276247288, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1, perItem = 442, counter:533341
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem = 416
> [TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276246856, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1, perItem = 416, counter:533342
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)